Spark program to read data from RDBMS

I wanted to figure out how to connect to RDBMS from spark and extract data, so i followed these steps. You can download this project form github
First i did create Address table in my local mysql like this

CREATE TABLE `address` (
  `addressid` int(11) NOT NULL AUTO_INCREMENT,
  `contactid` int(11) DEFAULT NULL,
  `line1` varchar(300) NOT NULL,
  `city` varchar(50) NOT NULL,
  `state` varchar(50) NOT NULL,
  `zip` varchar(50) NOT NULL,
  `lastmodified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`addressid`),
  KEY `contactid` (`contactid`),
  CONSTRAINT `address_ibfk_1` FOREIGN KEY (`contactid`) REFERENCES `CONTACT` (`contactid`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
Then i did add 5 sample records to the address table. When i query address table on my local this is what i get
After that i did create a Spark Scala project that has mysql-connector-java as one of the dependencies The last step was to create a simple Spark program like this, My program has 4 main sections
  1. First is Address as case class with same schema as that of Address table, without lastmodified field
  2. Next is this call to create object of JdbcRDD that says query everything from address with addressid between 1 and 5. new JdbcRDD(sparkContext, getConnection, "select * from address limit ?,?", 0, 5, 1, convertToAddress)
  3. Then i did define getConnection() method that creates JDBC connection to my database and returns it
  4. Last is the convertToAddress() method that knows how to take a ResultSet and convert it into object of Address
When i run this program in IDE this is the output i get

25 comments:

Philips Huges said...


Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

Installment loans
Payday loans
Title loans

Philips Huges said...

Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

Personal Installment Loans
Payday Cash Advance loan
Title Car loan
Cash Advance Loan

rithkhmer said...

Thank you very much for sharing such a beautiful article.
หนังออนไลน์

Jason Borne said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully. Call on our Toll Free no. 1 855 966 3855
Gmail Customer service is a third party technical support service for Gmail users when they face any technical issue or error in their Gmail account. Our Gmail Customer Support team solves issues like forgot Gmail account password, Gmail configuration or Sync issues, recover deleted emails and many more. Toll Free number (800) 986-9271
How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office com setup products are included in each plan and then follow the steps for your product. The steps below also apply if you're installing a single, stand-alone Office application such as Access 2016 or Visio 2016. Need Help with office setup Enter Product Key? Call 1-800-000-0000 Toll Free
Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. We offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices. Call our Toll Free number 1 855 966 3855
Other Services
Norton Toll Free , Office-Setup , office.com/setup.

Sowmiya said...

Really Good blog post.provided a helpful information.I hope that you will post more updates like this.
MSBI Training in Chennai

john wick said...
This comment has been removed by the author.
Jim said...

Nice article, thanks for sharing.
best Thank you, that’s great
طراحی سایت
طراحی وب سایت
سئو
طراحی سایت شرکتی
طراحی سایت ارزان
طراحی سایت فروشگاهی
طراحی سایت
طراحی وب سایت
سئو
طراحی سایت

Samatha Ravula said...

Iam very impressive your site gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you..

python online training

Smith said...

Online Supports...
norton.com/setup
mcafee.com/activate
office.com/setup
office.com/setup
mcafee.com/activate

sunitha vishnu said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
Android Training in Chennai
Ios Training in Chennai

paul william said...

mcafee.com/activate have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.
mcafee.com/activate have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.

roku.com/link provides the simplest way to stream entertainment to your TV. On your terms. Visit roku.com/link for Roku code linking to your Account.
roku.com/link provides the simplest way to stream entertainment to your TV. On your terms. Visit roku.com/link for Roku code linking to your Account.

john wick said...

Norton is one of the most reliable antivirus providers in the market. Norton is delivering top rated protection from mobile devices and computers. Norton has 3 antivirus packages
norton.com/setup
Microsoft office setup is the software setup file with this setup file you can install on your computer and some of the supported device to use Microsoft office.
www.office.com/setup
McAfee have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.
mcafee.com/activate
mcafee card activate
If your PC is having a slow performance and you are facing problems due to this, then we can provide a solution to that. at TechSalvation, we have a highly experienced team for resolving the issues related to the slow performance of computers and laptops.
avira tech support
Our Avira Customer Support is open to those who have issues relating to downloading, updating or upgrading Avira Antivirus software. We can also assist you to confirm your product if you are not sure about its validity.
avira support number
Our Avira Refund Support is open to those who have issues relating to downloading, updating or upgrading Avira Antivirus software. We can also assist you to confirm your product if you are not sure about its validity.
avira refund

Anonymous said...

www.Norton.com/Setup
Norton Internet Security | Ransomware protection
Get protection from Cyberattacks, Malware, trojan and all other online threats ,Get Norton setup with the help of product with get in touch with Norton Support Expert Norton.com/Setup


Anonymous said...

www.McAfee.com/Activate
Protect Your Computer , Network, Social Media Account And all other from hackers, infection ,Virus and other online threats, Mcafee Total Protection Software very Important to Activate or Setup with the Official Link McAfee.com/Activate

Facing Issues With Antivirus? McAfee Supportt Is Just A Call Away with the following link mcAfee.com/Activate to get Total protection Online instant Help

Anonymous said...

McAfee Support

McAfee Customer Service

McAfee Antivirus Support

McAfee Technical Support

McAfee Help

McAfee phone number

McAfee Customer Support

bella bason said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully.We at Avast Tech Support provides service to protect your PC from potential online threats and external attacks like viruses, Trojans, malwares, spywares and phishing scams. And Avast Refund. Call on our Avast Phone Number.


Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. At Norton Support we offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices.


The Antivirus Package protects one device and works on IOS, Windows, Android and MAC. It includes real time protection, virus, spyware, Malware and phishing protection.

Norton.com/setup
Norton.com setup
Norton setup

Smith said...

Online Supports...
norton.com/setup
office.com/setup
office.com/setup

Smith said...

Online Supports...
microsoft outlook support
adobe support
aol mail support


bella bason said...

Avast Customer Support is a third party technical support service. We are here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast setup related issues but will guide with how to get started with your new Avast product once it gets installed successfully. We at Avast Tech Support provides service to protect your PC from potential online threats and external attacks like viruses, Trojans, malwares, spywares and phishing scams.

http://avastcomsetup.com

Avast Support

bella bason said...

Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. At Norton Support we offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices.

http://supportfornorton.com

Norton Support

bella bason said...

The Antivirus Package protects one device and works on IOS, Windows, Android and MAC. It includes real time protection, virus, spyware, Malware and phishing protection. The Standard Package has the same features like the Antivirus Package plus secure pass protection ad additional protection option

http://nortoncomsetup-norton.com

Norton.com/setup

faraz rental said...

great
its so useful and helpful
very kind of you
اجاره خودرو
اجاره ماشین
اجاره خودرو تهران

mcafee.com activate said...

McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.


Install Office said...

We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.


office.com setup said...



MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.