Q:

Write a way to optimize your Java app that migrates millions of records from MariaDB database to SOLR server ?

0

Write a way to optimize your Java app that migrates millions of records from MariaDB database to SOLR server ?

The integration flow contains the following components:

1-A simple java application, entry point for CLI Execution

2-A java class which acts as a rest client

3-A Servlet, which uses the service class

4-A service class which implements the reading of the records from MariaDB and Insert/Update into SOLR index.

 

All Answers

need an explanation for this answer? contact us directly to get an explanation for this answer

The flow implemented is as follows:

Java App > Rest Client > Servlet > Service > DAO etc

public final class BatchSolrApp {
  private BatchSolrApp() {
  //I am capturing start time using localtime API
   BatchSolrClient batchSolrIndexingClient = new BatchSolrClient();
    int statusCode = batchSolrIndexingClient.updateSolrIndex();
public static void main(String[] args) {
        try {
            new BatchSolrApp();
        } catch (Exception exception) {
           //log.error('xxx'+exception.getMessage());
        }
    }
}
/**
 * API Client to call batch solr reindex service via servlet.
 */
public class BatchSolrClient {
    public int updateSolrIndex() {
        HttpClient httpClient = httpLibCustom.createProvider(loggerHttpClient).createHttpClient();        
        String xxxUri = "www.xxx.org/aaa/bbbb?action=batchIndex&csrfToken=fakeCsrfToken";        
        HttpUrl xxxHttpUri = HttpUrl.parse(xxxUri);
        Request request = new Request.Builder().url(xxxHttpUri).get().build();
        HttpAPIResponse httpAPIResponse = null;
        Response response = null;        
        try {
            httpAPIResponse = httpClient.executeHttpRequest(request);
            if (httpAPIResponse != null) {
                response = httpAPIResponse.getResponse();
                if (response != null) {
                    return response.code();
                } 
            }
        } catch (
                Exception excResponse) {
            logger.error("Exception xxxxxx : \n" +
                    exceptionStackTraceToString(excResponse));
            return 206;
        }
        return 202;
    }        
}
public class BatchSolrServlet extends HttpServlet {
    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException {
        HashMap<String, String> params = SomeServletUtils.getServletParams(req,
                resp, this);                
        ServletUtils.processRequest(this, req, resp, () -> new BatchSolrService(params));
    }
    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException {
        doPost(req, resp);
    }
}
/**
 * Service to execute Batch Solr Indexing
 */
public class BatchSolrService extends SomeAbstractClass {
public BatchSolrIndexingService(HashMap<String, String> params) {
         processRequest();
    }
 protected AbstractResponse processRequest() throws ServerException {
        CustomJSONResponse response = new CustomJSONResponse(this.getFunction());
                try {                  
                    List<String> productIdList = fetchAllProductIds();
                    if (productIdList != null && !productIdList.isEmpty()) {
                        //This read the records from MariaDB, creates SOLR docs from them and insert/add to SOLR 
                        //Next line Uses the following logic amongst other code
                        //EmbeddedSolrServer server = SolrService.getSolrServer();
                        // server.add(doc)
                        boolean batchUpdateOutcome = XXXSolrService.addProductBatchToSolr(productIdList);
                    } 
                } catch (Exception batchIndexException) {
                    logger.error("XXX " +exceptionStackTraceToString(batchIndexException));
                }
                response.setStatusCode(201);
                response.setSuccess();
                return response;       
        super.setError("Error during batch solr indexing request handling.", 400);
        return errorResponse;
    }    
}

o

need an explanation for this answer? contact us directly to get an explanation for this answer

total answers (1)

Similar questions


need a help?


find thousands of online teachers now