Spark program to read data from RDBMS

I wanted to figure out how to connect to RDBMS from spark and extract data, so i followed these steps. You can download this project form github
First i did create Address table in my local mysql like this

CREATE TABLE `address` (
  `addressid` int(11) NOT NULL AUTO_INCREMENT,
  `contactid` int(11) DEFAULT NULL,
  `line1` varchar(300) NOT NULL,
  `city` varchar(50) NOT NULL,
  `state` varchar(50) NOT NULL,
  `zip` varchar(50) NOT NULL,
  `lastmodified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`addressid`),
  KEY `contactid` (`contactid`),
  CONSTRAINT `address_ibfk_1` FOREIGN KEY (`contactid`) REFERENCES `CONTACT` (`contactid`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
Then i did add 5 sample records to the address table. When i query address table on my local this is what i get
After that i did create a Spark Scala project that has mysql-connector-java as one of the dependencies The last step was to create a simple Spark program like this, My program has 4 main sections
  1. First is Address as case class with same schema as that of Address table, without lastmodified field
  2. Next is this call to create object of JdbcRDD that says query everything from address with addressid between 1 and 5. new JdbcRDD(sparkContext, getConnection, "select * from address limit ?,?", 0, 5, 1, convertToAddress)
  3. Then i did define getConnection() method that creates JDBC connection to my database and returns it
  4. Last is the convertToAddress() method that knows how to take a ResultSet and convert it into object of Address
When i run this program in IDE this is the output i get

47 comments:

Philips Huges said...


Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

Installment loans
Payday loans
Title loans

Philips Huges said...

Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

Personal Installment Loans
Payday Cash Advance loan
Title Car loan
Cash Advance Loan

rithkhmer said...

Thank you very much for sharing such a beautiful article.
หนังออนไลน์

Jason Borne said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully. Call on our Toll Free no. 1 855 966 3855
Gmail Customer service is a third party technical support service for Gmail users when they face any technical issue or error in their Gmail account. Our Gmail Customer Support team solves issues like forgot Gmail account password, Gmail configuration or Sync issues, recover deleted emails and many more. Toll Free number (800) 986-9271
How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office com setup products are included in each plan and then follow the steps for your product. The steps below also apply if you're installing a single, stand-alone Office application such as Access 2016 or Visio 2016. Need Help with office setup Enter Product Key? Call 1-800-000-0000 Toll Free
Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. We offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices. Call our Toll Free number 1 855 966 3855
Other Services
Norton Toll Free , Office-Setup , office.com/setup.

Sowmiya said...

Really Good blog post.provided a helpful information.I hope that you will post more updates like this.
MSBI Training in Chennai

john wick said...
This comment has been removed by the author.
Jim said...

Nice article, thanks for sharing.
best Thank you, that’s great
طراحی سایت
طراحی وب سایت
سئو
طراحی سایت شرکتی
طراحی سایت ارزان
طراحی سایت فروشگاهی
طراحی سایت
طراحی وب سایت
سئو
طراحی سایت

Samatha Ravula said...

Iam very impressive your site gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you..

python online training

Smith said...

Online Supports...
norton.com/setup
mcafee.com/activate
office.com/setup
office.com/setup
mcafee.com/activate

sunitha vishnu said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
Android Training in Chennai
Ios Training in Chennai

paul william said...

mcafee.com/activate have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.
mcafee.com/activate have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.

roku.com/link provides the simplest way to stream entertainment to your TV. On your terms. Visit roku.com/link for Roku code linking to your Account.
roku.com/link provides the simplest way to stream entertainment to your TV. On your terms. Visit roku.com/link for Roku code linking to your Account.

john wick said...

Norton is one of the most reliable antivirus providers in the market. Norton is delivering top rated protection from mobile devices and computers. Norton has 3 antivirus packages
norton.com/setup
Microsoft office setup is the software setup file with this setup file you can install on your computer and some of the supported device to use Microsoft office.
www.office.com/setup
McAfee have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.
mcafee.com/activate
mcafee card activate
If your PC is having a slow performance and you are facing problems due to this, then we can provide a solution to that. at TechSalvation, we have a highly experienced team for resolving the issues related to the slow performance of computers and laptops.
avira tech support
Our Avira Customer Support is open to those who have issues relating to downloading, updating or upgrading Avira Antivirus software. We can also assist you to confirm your product if you are not sure about its validity.
avira support number
Our Avira Refund Support is open to those who have issues relating to downloading, updating or upgrading Avira Antivirus software. We can also assist you to confirm your product if you are not sure about its validity.
avira refund

Anonymous said...

www.Norton.com/Setup
Norton Internet Security | Ransomware protection
Get protection from Cyberattacks, Malware, trojan and all other online threats ,Get Norton setup with the help of product with get in touch with Norton Support Expert Norton.com/Setup


Anonymous said...

www.McAfee.com/Activate
Protect Your Computer , Network, Social Media Account And all other from hackers, infection ,Virus and other online threats, Mcafee Total Protection Software very Important to Activate or Setup with the Official Link McAfee.com/Activate

Facing Issues With Antivirus? McAfee Supportt Is Just A Call Away with the following link mcAfee.com/Activate to get Total protection Online instant Help

Anonymous said...

McAfee Support

McAfee Customer Service

McAfee Antivirus Support

McAfee Technical Support

McAfee Help

McAfee phone number

McAfee Customer Support

bella bason said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully.We at Avast Tech Support provides service to protect your PC from potential online threats and external attacks like viruses, Trojans, malwares, spywares and phishing scams. And Avast Refund. Call on our Avast Phone Number.


Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. At Norton Support we offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices.


The Antivirus Package protects one device and works on IOS, Windows, Android and MAC. It includes real time protection, virus, spyware, Malware and phishing protection.

Norton.com/setup
Norton.com setup
Norton setup

Smith said...

Online Supports...
norton.com/setup
office.com/setup
office.com/setup

Smith said...

Online Supports...
microsoft outlook support
adobe support
aol mail support


bella bason said...

Avast Customer Support is a third party technical support service. We are here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast setup related issues but will guide with how to get started with your new Avast product once it gets installed successfully. We at Avast Tech Support provides service to protect your PC from potential online threats and external attacks like viruses, Trojans, malwares, spywares and phishing scams.

http://avastcomsetup.com

Avast Support

bella bason said...

Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. At Norton Support we offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices.

http://supportfornorton.com

Norton Support

bella bason said...

The Antivirus Package protects one device and works on IOS, Windows, Android and MAC. It includes real time protection, virus, spyware, Malware and phishing protection. The Standard Package has the same features like the Antivirus Package plus secure pass protection ad additional protection option

http://nortoncomsetup-norton.com

Norton.com/setup

faraz rental said...

great
its so useful and helpful
very kind of you
اجاره خودرو
اجاره ماشین
اجاره خودرو تهران

mcafee.com activate said...

McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.


Install Office said...

We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.


office.com setup said...



MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

Rohit Sharma said...

Webroot Support - Call 1-844-456-8733 (US/CA) for Webroot Installation, Activation, Configuration, Billing, Renewal or any other queries. Contact our Webroot technical support to get Best Webroot Customer Service
webroot support

Rohit Sharma said...

dell 1-844-456-8733 (US/CA) any Installation, Drivers, Repair, Errors and many more. Get Online Resolution anytime by Technical Support.
dell printer support

Rohit Sharma said...

Epson Printer Support - Get Quick Setup for Wireless Epson Printer Install by calling
Epson Printer Support number 1-844-456-8733 (US/CA) Toll-free.
Epson Printer Support

Rohit Sharma said...

Keyword- hp printer support, hp printer support number, hp printer support phone number
HP Printer Support - Call HP Technical Support Phone Number 1-844-456-8733 (US/CA) or 01444 39 0119 (UK) for any HP Printer related issues.
Website- hp Printer Support

paul william said...

www.mcafee.com/activate
www.mcafee.com/activate
www.mcafee.com/activate
roku.com/link

john wick said...

Microsoft office setup is the software setup file with this setup file you can install on your computer and some of the supported device to use Microsoft office.

www.office.com/setup

thesis writing service said...

Thanks for sharing amazing information !!!!!!
Please keep up sharing.

bella shah said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully.We at Avast Tech Support provides service to protect your PC from potential online threats and external attacks like viruses, Trojans, malwares, spywares and phishing scams. And Avast Refund. Call on our Avast Phone Number.

Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. At Norton Support we offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices.

Norton.com/setup
Norton.com setup
Norton setup

office setup com said...


Before you plan to install the Office 2016 or Office 365 on your device be
it a Computer, Laptop, Mobile Phone or a Tablet, you are required to take
few important steps on of them is to remove any existing Office
installations from your PC. Just like the previous Office products, Office
2016 & 365 will conflict with the previously installed versions. So, it
becomes necessary to remove the previous office files properly.



setup.office.com
office.com
www.office.com/setup

www.office.com setup said...


To Setup retail card please visit official website.
Office Retail Cards allow you to download your security product from the
internet instead of installing from a CD, ensuring recent versions.

www.office.com/myaccount
office com setup
www.office.com setup

male extra pills said...

There is numerous separate years Los angeles Weight reduction eating plan with each a person is a necessity. The pioneer part can be your original getting rid of belonging to the extra pounds. la weight loss

aazim amixxxsuhaib xxkhan said...

Office includes word processing, spreadsheet, presentation graphics and email communication programs that provide functionality.
www.office.com/setup

aazim amixxxsuhaib xxkhan said...

AOL Email Support is one of the best email service providers that helps in AOL email related issues.There are several email providers like gmail, Hotmail, rediff and so on but AOL is the only company that provides the best service all over the world. aol related issues.
aol email support

www.norton.com/setup said...

Norton is the One of the best antivirus company all over world. It is an anti malware software developed and distributed by Symantec corporation.Norton setup uses heuristics and signatures to detect viruses in your computer. It features also include phishing protection and email spam filtering .Norton setup product is distributed as box copy, a download and as an OEM software.
www.norton.com/setup

mcafee.com/activate said...

McAfee have the complete set of features which can protect your digital online offline computing devices that increase the speed with inbuilt PC Optimization tool.
mcafee.com/activate

Abrar Hazaree said...

quick book is an accounting software for small and medium sized businesses where we can manage business payments, pay bills and pay rolls
quickbook support

Abrar Hazaree said...

Microsoft office provides quality of products like word, excel, power point, access, outlook, one drive etc
www.office.com/setup

john macla said...


webroot is the best spyware blocking and removel product called webroot spy sweeper. webroot also provides mobile device security.
www.webroot.com/safe

john macla said...

yahoo is search engine and web service provider.its also known for its web portal.yahoo provides business services also like yahoo web hosting.
yahoo support

mcafee.com/activate said...

Mcafee.com/activate Online Help – Step by Step guide for McAfee Activate, Download & complete installation online. We are providing independent support service if in case you face problem to activate or Activate McAfee product. Just fill the form below and will get in touch with you as quick as possible.

mcafee.com/activate

Office Setup said...

Office.com/setup - Instructions for Office Setup Installation with the help of this Blog. Get the installation help for Microsoft Office Follow the bearing on the page. you can download and introduce Office, Help with the installation process of Windows 10, Installation process for Office 365 Home
For Installation Help Please Visit...
www.office.com/setup

Roku Link Activate said...

Visit Roku Com Activate
AND ROKU COM SUPPORT AND GET PROFESSIONAL SUPPORT FROM YOUR CERTIFIED TECHNICIAN