Implement (Install, Configure & Test) a full, scalable solution of a Cloud server-based Hadoop Cluster with MongoDB (including Interface modules)

**PLEASE READ THE ATTACHED PDF AS IT PROVIDES FULL SPECS & REQUIREMENTS & DELIVERABLES FOR THIS PROJECT** This project entails a middle-tier and backend implementation of Hadoop and MongoDB. This Hadoop MongoDB implementation will then contain an ever-expanding database of genetic data. The database will be extremely large, starting out as terabytes and then expanding to petabytes. The data will be accessible to our end users from a web interface. We have already coded an API layer in C# that will interface with your module and invoke it, await the data and return the data to the client. Our web frontend is implemented as Drupal frontend, the API layer is C# .NET with a SQL server backend. Since this project is to develop a module of Hadoop and MongoDB, the currently used database technologies (ie SQL server) are not required for discussion. This module will be named as "HMM" for brevity. Goal The goal of HMM is to provide a superfast module for C# which implements Hadoop and MongoDB for database queries. Since Hadoop and MongoDB have some similarities we will explain later in this document how each one will be used in-order to create ultra-fast database querying. The data for the database will be structured and semi structured text files. These files will be uploaded from several different user types, including system admins and also end users. Requirements • Provide a clear implementation plan for both Hadoop and MongoDB. o Include recommended optimal cloud server specs for initial HMM implementation that will enable efficient scalability • Provide a workflow diagram that clearly identifies which components will be used and how each component will interface with each other. • Perform all server software installation for HMM. This includes but is not limited to a) Full installation and configuration of Hadoop module b) Full installation and configuration of MongoDB modules. • Provide a C# module that will interface with MongoDB. The aim of this interface module will be to provide an interface to execute queries. •Implement (installing, set-up and fully configure) a Hadoop cluster that uses Hadoop Streaming • Implement a straight-forward way to add more nodes into a Hadoop cluster (that we host). • MongoDB and Hadoop must be implemented in a manner that allows us to connect our source code BitBucket repository and push the content on to any new servers where we want a new instance of MongoDB and Hadoop and/or add and remove nodes from the cluster. Specifications • Server will be cloud-based server. • OS Support: Ubuntu 14+ for 64bit • OS Support: Windows 2012+ 64bit • Visual Studio solutions where required. • Install and configure Hadoop Cluster and install & configure MongoDB using the latest open source modules for each • Implement security features for Hadoop Cluster and MongoDB. • Install MongoDB on Windows or Linux depending on our discussion. • MongoDB module to be coded in C#, using the database drivers provided by Microsoft or by an approved third party. Delivered as working Visual Studio solution. •Hadoop C# module to interface with Hadoop, this should be preferably developed in C#. (If this is not possible then we would be open to alternatives.) • Implement Hadoop Distributes File System (DFS) query system. • Data in and data out should be JSON. Deliverables o Fully configured, extensible bug free installation of MongoDB. o Fully configured, extensible bug free installation of Hadoop cluster(s). o Fully configured, extensible bug free installation of Hadoop, Hadoop DFS and Hadoop Streaming. o Scalable, extensible, commented, bug free, working source code for the entire HMM. o Compiled binaries, HTML and other supporting files o Technical support o Testing Our company will own all source code, compiled code and intellectual property. Prior to engagement a signed confidentiality and IP assignment agreement will be required. **FULL INFO APPEARS IN THE ATTACHED PDF DOCUMENT - PLEASE READ**