tag:blogger.com,1999:blog-87064162867574647432024-03-18T15:23:24.759+07:00SAP Basis HowtoBlog about SAP ERP especially on technical field such as SAP Basis/NetWeaver concepts, monitoring, administration, performance tuning, SAP Implementations, ABAP, Data Migration, OS, Database server, and FAQsAnonymoushttp://www.blogger.com/profile/03193724019489170899noreply@blogger.comBlogger250125tag:blogger.com,1999:blog-8706416286757464743.post-22868040319768600182016-12-03T08:46:00.001+07:002016-12-03T08:46:13.656+07:00SAP Basis & Virtualization ServicesHi guys,<br />
<br />
I am offering Remote SAP Basis Support and Service to
your company. No matter your company have dedicated SAP Basis or not, we
can help you. Your company could hire me as Remote SAP Basis.<br />
<br />
I
also have virtualization support on VMware, Proxmox VE, Linux, Oracle,
and DB2. Usually, I support my client via TeamViewer connection.<br />
<br />
You can contact me through<br />
<br />
Email : <a data-mce-href="mailto:devratt@yahoo.com" href="mailto:devratt@yahoo.com" target="_blank">devratt@yahoo.com</a><br /> Phone/SMS/WA/Telegram : +62-85330356073Anonymoushttp://www.blogger.com/profile/03193724019489170899noreply@blogger.com183tag:blogger.com,1999:blog-8706416286757464743.post-63693017004476761132016-09-29T15:26:00.000+07:002016-09-29T15:26:04.808+07:00WordPress Development Best Practices and Tips<div class="content_body has-bottom_margin" data-view="content#body" style="border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; margin: 0px 0px 20px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; box-sizing: border-box; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
WordPress is easily the most powerful open source blogging and content management system available online today, and so working knowledge of its intricacies is a boon to any <a href="https://www.toptal.com/designers/freelance/portfolios#portfolio-guide">developer or designer resume</a>.</div>
<div style="border: 0px; box-sizing: border-box; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This resource contains a collection of WordPress development best practices and WordPress development tips provided by our Toptal network members. As such, this page will be updated on a regular basis to include additional information and cover emerging techniques. This is a community driven project, so you are encouraged to contribute as well, and we are counting on your feedback.</div>
<div style="border: 0px; box-sizing: border-box; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Check out the Toptal <a href="https://www.toptal.com/resources" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">resource pages</a> for additional information on WordPress development; there is a WordPress developer <a href="https://www.toptal.com/wordpress#hiring-guide" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">hiring guide</a> and WordPress developer <a href="https://www.toptal.com/wordpress/interview-questions" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">interview questions</a>.</div>
</div>
<div class="qa-tip" style="border: 0px; box-sizing: border-box; font-family: "Proxima Nova", Arial, sans-serif; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="content_body has-bottom_margin" data-view="content#body" style="border: 0px; box-sizing: border-box; margin: 0px 0px 20px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<h2 style="border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Use Child Themes and Plugins</span></h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A lot of newcomers to WordPress dive right in and start modifying their core theme files. This is a definite mistake. All of your changes will disappear right after an upgrade, and since plugins and themes are updated about as often as apps on your phone, this is pretty frequently.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To avoid this, create children of your plugins and themes. Not only will you preserve your changes, you can upgrade on your own time. The same steps used to create a child theme can be applied to creating a child plugin, but let’s use creating a child theme as our example.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get started making your child theme, create a new folder in your themes folder with a unique name, then create a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">style.css</code> file in your new folder.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In WordPress, all theme parameters are stored in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">style.css</code> file as the first comment block. Open the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">style.css</code> from your original theme, the parent, to see an example of this.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For your child theme, go ahead and copy that comment block from your original theme’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">style.css</code> to the new file and rename it. Adding the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">template</code> parameter to this header will link your new theme to the original. The Template parameter should point to the folder name of the original theme, like so:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-css hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/*
Theme Name: Twenty Fourteen Child Theme
Description: My new child theme based on Twenty Fourteen
Template: twentyfourteen
Version: 1.0
*/</span>
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, if you want to modify files of the original theme, copy them from the original theme’s folder and paste them into your child theme folder. WordPress will use original template files unless it sees the same file in your child theme. So if you want to make changes to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">header.php</code>, copy it from your original theme into your new child theme folder, and make the changes in the copy. For adding new or modified code, you likewise create a new functions file in your child theme and make your changes there.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This same file copying strategy goes for many plugins as well: create a folder with the same name as a plugin inside your child theme, and then adhering to the file structure of the original plugin folder, copy files from the original plugin to your new folder and modify them. Most major plugins support this, but it’s always good to check with the author if you are not sure.</div>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Speed Things Up with Caching</span></h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
WordPress optimized hosting services such as <a href="https://www.siteground.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Siteground</a>, or the more expensive <a href="https://wpengine.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Wpengine</a>, automatically support WordPress caching. If your host is one that has WordPress specific caching available, it’s the best option.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For those running on a VPS server with root access, Google PageSpeed is a turn key caching and optimization solution by Google that works with Apache and nginx. If that’s of interest to you, check out this guide on how to <a href="https://fabriceleven.com/dev/how-to-install-google-pagespeed-in-parallels/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">install PageSpeed on Plesk with CentOS</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If all of that sounds like too much work, then go with Cloudflare, a free CDN/Firewall/Caching and minification system.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Speaking of minification, minify your files yourself during development. Third party tools tend to break things more often than not, especially during upgrades. Doing it yourself gives you more control and awareness of when and where things go wrong.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Pay Attention to Security</span></h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
WordPress’s popularity makes it a high priority target for hackers. If you don’t update often, you are pretty much asking to get your site hacked.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Automatic updates are a little too dangerous for users with a lot of customizations and plugins, which is why I strongly suggest installing some sort of security plugin.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I personally recommend <a href="https://wordpress.org/plugins/better-wp-security/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">iTheme Security</a>, which implements security options like a password lockout and file monitoring. And <a href="https://wordpress.org/plugins/wordfence/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Wordfence Security</a>, a WordPress specific firewall for your site.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Three Developer Tools to Make Life Easier</span></h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
WordPress has many plugins and add-ons to make your developer life a lot easier. In particular, I recommend:</div>
<h4 id="wp-cli" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
WP-Cli</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://wp-cli.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">WP-Cli</a> lets you work with WordPress using the command line. With this great tool you can upgrade and downgrade WordPress in seconds, as well as update plugins. Notably, when you find yourself migrating to a different server, the built in search-replace command will take care of all the url changes for you, and it’s worth installing it simply because of that.</div>
<h4 id="advanced-database-cleaner" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Advanced Database Cleaner</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://wordpress.org/plugins/advanced-database-cleaner/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Advanced Database Cleaner</a> plugin cleans out spam comments, built in revisions, and transients. You can even set up tasks to run automatically.</div>
<h4 id="query-monitor" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Query Monitor</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When things are running slowly and you’re not sure what to blame, <a href="https://wordpress.org/plugins/query-monitor/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Query Monitor</a> lets you see what queries are taking too long to execute, as well as show you PHP warnings and errors.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Don't Overdose on Plugins</span></h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Yes, WordPress has tons of plugins, but that doesn’t mean you should install them all. The more plugins you have, the bulkier your site and the slower your loading times, so don’t use plugins unless absolutely necessary. If you only need to add a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">few</em> custom fields to your posts (a functionality already included in WordPress) don’t overengineer the solution by installing the advanced custom field plugin, <a href="https://wordpress.org/plugins/advanced-custom-fields/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ACF</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you must use a lot of plugins, make sure you have <a href="https://wordpress.org/plugins/plugin-organizer/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Plugin Organizer</a> installed to manage them. This great plugin lets you specify what plugins are activated on which pages (you can even use regular expressions), and this selective loading will significantly speed up your site.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can also use tools like <a href="https://wordpress.org/plugins/p3-profiler/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">P3 (Plugin Performance Profiler)</a> to see what plugins are taking up most of your precious resources.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Spring Clean Your WordPress Functions</span></h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Although great, WordPress comes out of the box with a lot of things that cannot be turned off in the settings. Here are a handful of steps to take any fresh WordPress install and make it more secure and perform better:</div>
<h3 id="remove-the-wordpress-version" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Remove the WordPress Version</span></h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Get rid of the WordPress version number to make your site harder to be identified by hackers. To do this, add the following to your functions.php file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">add_filter( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'the_generator'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'__return_null'</span> );
</code></pre>
<h3 id="remove-script-versions" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Remove Script Versions</span></h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Get rid of the version number after scripts. By default, WordPress adds versions to all your scripts. This can lead to issues with caching/minification plugins, as well as helps hackers identify your site better. To prevent this functionality, add the following code to your theme functions file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">remove_cssjs_ver</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">( <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span> )</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span>( strpos( <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'?ver='</span> ) )
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span> = remove_query_arg( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'ver'</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span> );
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span>;
}
add_filter( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'style_loader_src'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'remove_cssjs_ver'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1000</span> );
add_filter( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'script_loader_src'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'remove_cssjs_ver'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1000</span> );
</code></pre>
<h3 id="restrict-woocommerce" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Restrict WooCommerce</span></h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Did you install WooCommerce, and now the site is running slowly? Adding this function to your <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">functions.php</code>will prevent WooCommerce from loading its scripts on non-WooCommerce pages:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/**
* Tweak WooCommerce styles and scripts.
* Original credit goes to Greg from: https://gist.github.com/gregrickaby/2846416
*/</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">grd_woocommerce_script_cleaner</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Remove the generator tag, to reduce WooCommerce based hacking attacks</span>
remove_action( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wp_head'</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">array</span>( <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$GLOBALS</span>[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce'</span>], <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'generator'</span> ) );
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Unless we're in the store, remove all the scripts and junk!</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> ( ! is_woocommerce() && ! is_cart() && ! is_checkout() ) {
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce_frontend_styles'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-general'</span>);
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-layout'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-smallscreen'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce_fancybox_styles'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce_chosen_styles'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce_prettyPhoto_css'</span> );
wp_dequeue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'select2'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-add-payment-method'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-lost-password'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc_price_slider'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-single-product'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-add-to-cart'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-cart-fragments'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-credit-card-form'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-checkout'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-add-to-cart-variation'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-single-product'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-cart'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-chosen'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'prettyPhoto'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'prettyPhoto-init'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'jquery-blockui'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'jquery-placeholder'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'jquery-payment'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'jqueryui'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'fancybox'</span> );
wp_dequeue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wcqi-js'</span> );
}
}
add_action( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wp_enqueue_scripts'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'grd_woocommerce_script_cleaner'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">99</span> );
</code></pre>
<h3 id="enable-shortcodes-in-a-widget-area" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Enable Shortcodes in a Widget Area</span></h3>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Trying to use shortcodes in a widget area and getting nothing? Drop this in into <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">functions.php</code> to use shortcodes in widget areas:</span></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span style="font-size: small;">add_filter(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'widget_text'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'do_shortcode'</span>);</span></code></pre>
</h2>
<h2 style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Use a Function to Load Scripts and CSS</span></h2>
<h3 id="enable-shortcodes-in-a-widget-area" style="background-color: white; border: 0px; box-sizing: border-box; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">WordPress already keeps track of all the scripts and CSS that it has loaded, so instead of adding your JS and CSS into a header or footer, let WordPress handle it with its <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">enqueue</code> functionality. This way WordPress will keep dependencies in check and you will avoid potential conflicts. You add enqueue methods to your theme’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">function.php</code> file: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script()</code> or <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_style()</code>, respectively. Here is an example with some explanatory comments:</span></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span style="font-size: small;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">add_theme_scripts</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//example script from CDN, true means our script will be in the footer.</span>
wp_enqueue_script( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'particles'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'//cdn.jsdelivr.net/particles.js/2.0.0/particles.min.js'</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">array</span>(), <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span> );
});
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//All referred to when to load your style like: 'screen', 'print' or 'handheld.</span>
wp_enqueue_style( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'slider'</span>, get_template_directory_uri() . <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/css/slider.css'</span>,<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'all'</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//this will actually execute our function above</span>
add_action( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wp_enqueue_scripts'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'add_theme_scripts'</span> );
</span></code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;"><a href="https://developer.wordpress.org/themes/basics/including-css-javascript/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">WordPress.org’s Theme Handbook</a> further explains the many parameters to an enqueue method, but here’s the method signature for both enqueue methods:</span></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span style="font-size: small;">wp_enqueue_style(<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$handle</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$deps</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$ver</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$media</span> );
wp_enqueue_script(<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$handle</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$src</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$deps</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$ver</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$in_footer</span>);
</span></code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">As you can see, the only difference between these two methods is the final parameter. For <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_style()</code>, the last parameter sets the media for which this stylesheet has been defined. Such as <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">screen</code> (computer), <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">print</code> (print preview mode), <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">handheld</code>, and so on. The last parameter for <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script()</code> specifies whether or not the script should be loaded in the footer.</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Here’s a breakdown for the other parameters in our example:</span></div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$handle</code> is the name of the stylesheet, which can be anything you’d like.</span></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$src</code> is where the stylesheet is located (CDN, local, etc). This is the only required parameter.</span></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$deps</code> stands for dependencies. When passed a stylesheet handle, it will load it before executing the source script. When <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$deps</code> is passed in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script()</code>, it’s an array.</span></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$ver</code> sets the version number.</span></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$media</code> is the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_style()</code> only parameter. It specifies which type of display media the stylesheet is designed for, such as ‘all’, ‘screen’, ‘print’ or ‘handheld.’</span></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0px 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$in_footer</code> is the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script()</code>’s only parameter, a boolean that allows you to place your scripts in the footer of your HTML rather than in the header, which means it will not delay the loading of the DOM tree.</span></li>
</ul>
<div>
<span style="font-size: small;"><br /></span></div>
</h3>
<h3 class="tip_category_contributors-title" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; line-height: 28px; margin: 0px 0px 20px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: small;">Contributors:</span></h3>
<h3 style="background-color: white; border: 0px; box-sizing: border-box; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<ul class="tip_category_contributors-list" style="border: 0px; box-sizing: border-box; color: #303030; font-weight: normal; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="tip_category_contributor" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div class="tip_category_contributor-content" style="border: 0px; box-sizing: border-box; display: flex; flex-wrap: wrap; margin: 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="tip_category_contributor-photo_wrapper" style="-webkit-box-flex: 0; align-self: center; border: 0px; box-sizing: border-box; flex: 0 0 65px; height: 65px; margin: 0px 20px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 65px;"><span style="font-size: small;"><img alt="Small fe1da09f0218bd10ecb65b06b14a0547" class="tip_category_contributor-photo" src="https://assets.toptal.io/uploads/user/photo/203010/small_fe1da09f0218bd10ecb65b06b14a0547.jpg" style="border-radius: 4px; border: 0px; box-sizing: border-box; height: 65px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: auto;" /></span></span><span class="tip_category_contributor-info" style="-webkit-box-flex: 2; align-self: center; border: 0px; box-sizing: border-box; display: block; flex: 2 0 auto; line-height: 22px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;"><a href="https://www.toptal.com/designers/resume/alex-gurevich">Alex Gurevich</a>, <span style="color: #303030; font-weight: normal;">a <a href="https://www.toptal.com/wordpress/tips-and-practices">Toptal</a><a href="https://www.toptal.com/wordpress"> freelance developer</a></span></span><span class="tip_category_contributor-location" style="border: 0px; box-sizing: border-box; color: darkgrey; display: block; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: small;">Tucson, United States</span></span></span></div>
</li>
</ul>
</h3>
<h3 id="enable-shortcodes-in-a-widget-area" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></h3>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/14742130202178750860noreply@blogger.com325tag:blogger.com,1999:blog-8706416286757464743.post-56122175605960481802016-09-21T14:43:00.002+07:002016-09-21T14:43:40.242+07:00The 10 Most Common Mistakes That WordPress Developers Make<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We are only human, and one of the traits of being a human is that we make mistakes. On the other hand, we are also self-correcting, meaning we tend to learn from our mistakes and hopefully are thereby able to avoid making the same ones twice. A lot of the mistakes I have made in the WordPress realm originate from trying to save time when implementing solutions. However, these would typically rear their heads down the road when issues would crop up as a result of this approach. Making mistakes is inevitable. However, learning from other people’s oversights (and your own of course!) is a road you should proactively take.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="WordPress" src="https://assets.toptal.io/uploads/blog/image/121148/toptal-blog-image-1473346255146-be87e01c6f10f15e58358d278acbf2c5.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; text-align: center;">Engineers look like superheroes, but we’re still human. Learn from us.</span></div>
<div class="tweet_this" style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; margin: -10px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<br /></div>
<h2 id="common-mistake-1-keeping-the-debugging-off" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #1: Keeping the Debugging Off</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Why should I use debugging when my code is working fine? Debugging is a feature built into WordPress that will cause all PHP errors, warnings, and notices (about deprecated functions, etc.) to be displayed. When debugging is turned off, there may be important warnings or notices being generated that we never see, but which might cause issues later if we don’t deal with them in time. We want our code to play nicely with all the other elements of our site. So, when adding any new custom code to WordPress, you should always do your development work with debugging turned on (but make sure to turn it off before deploying the site to production!).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To enable this feature, you’ll need to edit the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp-config.php</code> file in the root directory of your WordPress install. Here is a snippet of a typical file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Enable debugging</span>
define(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'WP_DEBUG'</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Log all errors to a text file located at /wp-content/debug.log</span>
define(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'WP_DEBUG_LOG'</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Don’t display error messages write them to the log file /wp-content/debug.log</span>
define(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'WP_DEBUG_DISPLAY'</span>, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Ensure all PHP errors are written to the log file and not displayed on screen</span>
@ini_set(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'display_errors'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is not an exhaustive list of configuration options that can be used, but this suggested setup should be sufficient for most debugging needs.</div>
<h2 id="common-mistake-2-adding-scripts-and-styles-using-wphead-hook" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #2: Adding Scripts and Styles Using <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_head</code> Hook</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What is wrong with adding the scripts into my header template? WordPress already includes a plethora of <a href="https://developer.wordpress.org/themes/basics/including-css-javascript/#default-scripts-included-and-registered-by-wordpress" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">popular scripts</a>. Still, many developers will add additional scripts using the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_head</code> hook. This can result in the same script, but a different version, being loaded multiple times.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Enqueuing here comes to the rescue, which is the WordPress friendly way of adding scripts and styles to our website. We use enqueuing to prevent plugin conflicts and handle any dependencies a script might have. This is achieved by using the inbuilt functions <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script</code> or <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_style</code> to enqueue scripts and styles respectively. The main difference between the two functions is that with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_enqueue_script</code> we have an additional parameter that allows us to move the script into the footer of the page.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">wp_register_script( $handle, $src, $deps = array(), $ver = false, $in_footer = false )
wp_enqueue_script( $handle, $src = false, $deps = array(), $ver = false, $in_footer = false )
wp_register_style( $handle, $src, $deps = array(), $ver = false, $media = 'all' )
wp_enqueue_style( $handle, $src = false, $deps = array(), $ver = false, $media = 'all' )
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If the script is not required to render content above the fold, we can safely move it to the footer to make sure the content above the fold loads quickly. It’s <a href="https://www.toptal.com/wordpress/tips-and-practices" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">good practice</a> to register the script first before enqueuing it, as this allows others to deregister your script via the handle in their own plugins, without modifying the core code of your plugin. In addition to this, if the handle of a registered script is listed in the array of dependencies of another script that has been enqueued, that script will automatically be loaded prior to loading that highlighted enqueued script.</div>
<h2 id="common-mistake-3-avoiding-child-themes-and-modifying-wordpress-core-files" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #3: Avoiding Child Themes and Modifying WordPress Core Files</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Always create a child theme if you plan on modifying a theme. Some developers will make changes to the parent theme files only to discover after an upgrade to the theme that their changes have been overwritten and lost forever.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To create a child theme, place a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">style.css</code> file in a subdirectory of the child theme’s folder, with the following content:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-css hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/*
Theme Name: Twenty Sixteen Child
Theme URI: http://example.com/twenty-fifteen-child/
Description: Twenty Fifteen Child Theme
Author: John Doe
Author URI: http://example.com
Template: twentysixteen
Version: 1.0.0
License: GNU General Public License v2 or later
License URI: http://www.gnu.org/licenses/gpl-2.0.html
Tags: light, dark, two-columns, right-sidebar, responsive-layout, accessibility-ready
Text Domain: twenty-sixteen-child
*/</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The above example creates a child theme based on the default WordPress theme, <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Twenty Sixteen</em>. The most important line of this code is the one containing the word “Template” which must match the directory name of the parent theme you are cloning the child from.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The same principles apply to WordPress core files: Don’t take the easy route by modifying the core files. Put in that extra bit of effort by employing WordPress pluggable functions and filters to prevent your changes from being overwritten after a WordPress upgrade. Pluggable functions let you override some core functions, but this method is slowly being phased out and replaced with filters. Filters achieve the same end result and are inserted at the end of WordPress functions to allow their output to be modified. A trick is always to wrap your functions with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">if ( !function_exists() )</code> when using pluggable functions since multiple plugins trying to override the same pluggable function without this wrapper will produce a fatal error.</div>
<h2 id="common-mistake-4-hardcoding-values" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #4: Hardcoding Values</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Often it looks quicker to just hardcode a value (such as a URL) somewhere in the code, but the time spent down the road debugging and rectifying issues that arise as a result of this is far greater. By using the corresponding function to generate the desired output dynamically, we greatly simplify subsequent maintenance and debugging of our code. For example, if you migrate your site from a test environment to production with hardcoded URLs, all of a sudden you’ll notice your site it is not working. This is why we should employ functions, like the one listed below, for generating file paths and links:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Get child theme directory uri</span>
stylesheet_directory_uri();
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Get parent theme directory</span>
get_template_directory_uri();
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Retrieves url for the current site</span>
site_url();
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another bad example of hardcoding is when writing custom queries. For example, as a security measure, we change the default WordPress datatable prefix from <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_</code> to something a little more unique, like <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp743_</code>. Our queries will fail if we ever move the WordPress install, as the table prefixes can change between environments. To prevent this from happening, we can reference the <a href="https://codex.wordpress.org/wpdb#Tables" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">table properties</a> of the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wpdb</code> class:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">global</span> <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$wpdb</span>;
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$user_count</span> = <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$wpdb</span>->get_var( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"SELECT COUNT(*) FROM $wpdb->users"</span> );
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Notice how I am not using the value <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_users</code> for the table name, but instead, I’m letting WordPress work it out. Using these properties for generating the table names will help ensure that we return the correct results.</div>
<h2 id="common-mistake-5-not-stopping-your-site-from-being-indexed" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #5: Not Stopping Your Site From Being Indexed</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Why wouldn’t I want search engines to index my site? Indexing is good, right? Well, when building a website, you don’t want search engines to index your site until you have finished building it and have established a permalink structure. Furthermore, if you have a staging server where you test site upgrades, you don’t want search engines like Google indexing these duplicate pages. When there are multiple pieces of indistinguishable content, it is difficult for search engines to decide which version is more relevant to a search query. Search engines will in such cases penalize sites with duplicate content, and your site will suffer in search rankings as a result of this.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As shown below, <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">WordPress Reading Settings</em> has a checkbox that reads “Discourage search engines from indexing this site”, although this does have an important-to-note underneath stating that “It is up to search engines to honor this request”.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="WordPress Reading Settings" src="https://assets.toptal.io/uploads/blog/image/121146/toptal-blog-image-1473090685498-bd9b4071dbd51b9527b5428feb1792d7.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Bear in mind that search engines often do <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">not</em> honor this request. Therefore, if you want to <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">reliably</em> prevent search engines from indexing your site, edit your <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.htaccess</code> file and insert the following line:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Header set X-Robots-Tag "noindex, nofollow"
</code></pre>
<h2 id="common-mistake-6-not-checking-if-a-plugin-is-active" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #6: Not Checking if a Plugin is Active</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Why should I check if a plugin function exists if my plugin is always switched on? For sure, 99% of the time your plugin will be active. However, what about that 1% of the time when for some reason it has been deactivated? If and when this occurs, your website will probably display some ugly PHP errors. To prevent this, we can check to see if the plugin is active before we call its functions. If the plugin function is being called via the front-end, we need to include the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">plugin.php</code> library in order to call the function <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">is_plugin_active()</code>:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">include_once</span>( ABSPATH . <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wp-admin/includes/plugin.php'</span> );
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> ( is_plugin_active( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'plugin-folder/plugin-main-file.php'</span> ) ) {
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Run plugin code</span>
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This technique is usually quite reliable. However, there could be instances where the author has changed the main plugin directory name. A more robust method would be to check for the existence of a class in the plugin:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span>( class_exists( ‘WooCommerce’ ) ) {
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// The plugin WooCommerce is turned on</span>
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Authors are less likely to change the name of a plugin’s class, so I would generally recommend using this method.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptal" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-image: initial; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-right-color: initial; border-right-style: initial; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-image: initial; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-right-color: initial; border-right-style: initial; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="common-mistake-7-loading-too-many-resources" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #7: Loading Too Many Resources</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Why should we be selective in loading plugin resources for pages? There is no valid reason to load styles and scripts for a plugin if that plugin is not used on the page that the user has navigated to. By only loading plugin files when necessary, we can reduce our page loading time, which will result in an improved end user experience. Take, for example, a WooCommerce site, where we only want the plugin to be loaded on our shopping pages. In such a case, we can selectively remove any files from being loaded on all the other sites pages to reduce bloating. We can add the following code to the theme or plugin’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">functions.php</code> file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">load_woo_scripts_styles</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span>{</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span>( function_exists( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'is_woocommerce'</span> ) ){
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Only load styles/scripts on Woocommerce pages </span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span>(! is_woocommerce() && ! is_cart() && ! is_checkout() ) {
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Dequeue scripts.</span>
wp_dequeue_script(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce'</span>);
wp_dequeue_script(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-add-to-cart'</span>);
wp_dequeue_script(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wc-cart-fragments'</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Dequeue styles. </span>
wp_dequeue_style(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-general'</span>);
wp_dequeue_style(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-layout'</span>);
wp_dequeue_style(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce-smallscreen'</span>);
}
}
}
add_action( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'wp_enqueue_scripts'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'load_woo_scripts_styles'</span>);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Scripts can be removed with the function <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_dequeue_script($handle)</code> via the handle with which they were registered. Similarly, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">wp_dequeue_style($handle)</code> will prevent stylesheets from being loaded. However, if this is too challenging for you to implement, you can install the <a href="https://wordpress.org/plugins/plugin-organizer/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Plugin Organizer</a> that provides the ability to load plugins selectively based on certain criteria, such as a post type or page name. It’s a good idea to disable any caching plugins, like <a href="https://wordpress.org/plugins/w3-total-cache/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">W3Cache</a>, that you may have switched on to stop you from having to refresh the cache constantly to reflect any changes you have made.</div>
<h2 id="common-mistake-8-keeping-the-admin-bar" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #8: Keeping the Admin Bar</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Can’t I just leave the WordPress Admin Bar visible for everyone? Well, yes, you could allow your users access to the admin pages. However, these pages very often do not visually integrate with your chosen theme and don’t provide a seamless integration. If you want your site to look professional, you should disable the Admin Bar and provide a front-end account management page of your own:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">add_action(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'after_setup_theme'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'remove_admin_bar'</span>);
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">remove_admin_bar</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (!current_user_can(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'administrator'</span>) && !is_admin()) {
show_admin_bar(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>);
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The above code, when copied into your theme’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">functions.php</code> file will only display the Admin Bar for administrators of the site. You can add any of the WordPress <a href="https://codex.wordpress.org/Roles_and_Capabilities" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">user roles</a> or capabilities into the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">current_user_can($capability)</code> function to exclude users from seeing the admin bar.</div>
<h2 id="common-mistake-9-not-utilizing-the-gettext-filter" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #9: Not Utilizing the GetText Filter</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I can use CSS or JavaScript to change the label of a button, what’s wrong with that? Well, yes, you can. However, you’re adding superfluous code and extra time to render the button, when you can instead use one of the handiest filters in WordPress, called <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">gettext</code>. In conjunction with a plugin’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">textdomain</code>, a unique identifier that ensures WordPress can distinguish between all loaded translations, we can employ the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">gettext</code>filter to modify the text before the page is rendered. If you search the source code for the function <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">load_plugin_textdomain($domain)</code>, it will give you the domain name we need to override the text in question. Any reputable plugin will ensure that the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">textdomain</code> for a plugin is set on initialization of the plugin. If it’s some text in a theme that you want to change, search for the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">load_theme_textdomain($domain)</code> line of code. Using WooCommerce once again as an example, we can change the text that appears for the “Related Products” heading. Insert the following code into your theme’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">functions.php</code> file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">translate_string</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">( <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$translated_text</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$untranslated_text</span>, <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$domain</span> )</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> ( <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$translated_text</span> == <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Related Products'</span>) {
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$translated_text</span> = __( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Other Great Products'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce'</span> );
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$translated_text</span>;
}
add_filter( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gettext'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'translate_string'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">15</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span> );
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This filter hook is applied to the translated text by the internationalization functions <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">__()</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">_e()</code>, as long as the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">textdomain</code> is set via the aforementioned functions.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-php hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">_e( <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Related Products'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'woocommerce'</span> );
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Search your plugins for these internationalization functions to see what other strings you can customize.</div>
<h2 id="common-mistake-10-keeping-the-default-permalinks" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Common Mistake #10: Keeping the Default Permalinks</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By default, WordPress uses a query string with the post’s ID to return the specified content. However, this is not user-friendly and users may remove pertinent parts of the URL when copying it. More importantly, these default permalinks do not use a search engine friendly structure. Enabling what we call “pretty” permalinks will ensure our URLs contain relevant keywords from the post title to improve performance in search engine rankings. It can be quite a daunting task having to retrospectively modify your permalinks, especially if your site has been running for a significant period of time, and you’ve got hundreds of posts already indexed by search engines. So after you’ve installed WordPress, ensure you <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">promptly</em> change your permalinks structure to something a little more search engine friendly than just a post ID. I generally use the post name for the majority of sites I build, but you can customize the permalink to whatever format you like using the available<a href="https://codex.wordpress.org/Using_Permalinks" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">permalink structure tags</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="WordPress Permalink Settings" src="https://assets.toptal.io/uploads/blog/image/121145/toptal-blog-image-1473090647255-589ca439f39b957b0915a0a8156777aa.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 817px;" /></div>
<h2 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Conclusion</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article is by no means an exhaustive list of mistakes made by <a href="https://www.toptal.com/wordpress" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">WordPress developers</a>. If there’s one thing you should take away from this article, though, it’s that you should never take shortcuts (and that’s true in any development platform, not just in WordPress!). Time saved now by poor programming practices will come back to haunt you later. Feel free to share with us some mistakes that you have made in the past – and more importantly any lessons learned – by leaving a comment below.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/andrew-schultz" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Andrew Schultz</a>, a <a href="https://www.toptal.com/wordpress/top-wordpress-development-mistakes">Toptal</a><a href="https://www.toptal.com/wordpress"> freelance developer</a>.</div>
Anonymoushttp://www.blogger.com/profile/14742130202178750860noreply@blogger.com100tag:blogger.com,1999:blog-8706416286757464743.post-74996094205346154292016-09-15T16:26:00.002+07:002016-09-15T16:26:48.981+07:00How Sequel and Sinatra Solve Ruby’s API Problem<h2 id="introduction" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Introduction</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In recent years, the number of JavaScript single page application frameworks and mobile applications has increased substantially. This imposes a correspondingly increased demand for server-side APIs. With Ruby on Rails being one of the today’s most popular web development frameworks, it is a natural choice among many developers for creating back-end API applications.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Yet while the Ruby on Rails architectural paradigm makes it quite easy to create back-end API applications, using Rails <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">only</em> for the API is overkill. In fact, it’s overkill to the point that even the Rails team has recognized this and has therefore introduced a new API-only mode in version 5. With this new feature in Ruby on Rails, creating API-only applications in Rails became an even easier and more viable option.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But there are other options too. The most notable are two very mature and powerful gems, which in combination provide powerful tools for creating server-side APIs. They are <a href="http://www.sinatrarb.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Sinatra</a> and <a href="http://sequel.jeremyevans.net/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Sequel</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Both of these gems have a very rich feature set: Sinatra serves as the domain specific language (DSL) for web applications, and Sequel serves as the object-relational mapping (ORM) layer. So, let’s take a brief look at each of them.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="API With Sinatra and Sequel: Ruby Tutorial" src="https://assets.toptal.io/uploads/blog/image/121155/toptal-blog-image-1473794069465-2e0f2fa87fd142bd2f0b1b7e6240db43.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; text-align: center;">Ruby API on a diet: introducing Sequel and Sinatra.</span></div>
<h2 id="sinatra" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sinatra</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sinatra is Rack-based web application framework. The Rack is a well known Ruby web server interface. It is used by many frameworks, like Ruby on Rails, for example, and supports lot of web servers, like WEBrick, Thin, or Puma. Sinatra provides a minimal interface for writing web applications in Ruby, and one of its most compelling features is support for middleware components. These components lie between the application and the web server, and can monitor and manipulate requests and responses.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For utilizing this Rack feature, Sinatra defines internal DSL for creating web applications. Its philosophy is very simple: Routes are represented by HTTP methods, followed by a route matching a pattern. A Ruby block within which request is processed and the response is formed.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/'</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Hello from sinatra'</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The route matching pattern can also include a named parameter. When route block is executed, a parameter value is passed to the block through the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">params</code> variable.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/players/:sport_id'</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># Parameter value accessible through params[:sport_id]</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Matching patterns can use splat operator <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">*</code> which makes parameter values available through <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">params[:splat]</code>.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/players/*/:year'</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># /players/performances/2016</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># Parameters - params['splat'] -> ['performances'], params[:year] -> 2016</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is not the end of Sinatra’s possibilities related to route matching. It can use more complex matching logic through regular expressions, as well as custom matchers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sinatra understands all of the standard HTTP verbs needed for creating a REST API: Get, Post, Put, Patch, Delete, and Options. Route priorities are determined by the order in which they are defined, and the first route that matches a request is the one that serves that request.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sinatra applications can be written in two ways; using classical or modular style. The main difference between them is that, with the classical style, we can have only one Sinatra application per Ruby process. Other differences are minor enough that, in most cases, they can be ignored, and the default settings can be used.</div>
<h3 id="classical-approach" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Classical Approach</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Implementing classical application is straightforward. We just have to load Sinatra and implement route handlers:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'sinatra'</span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/'</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Hello from Sinatra'</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By saving this code to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">demo_api_classic.rb</code> file, we can start the application directly by executing the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ruby demo_api_classic.rb
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, if the application is to be deployed with Rack handlers, like Passenger, it is better to start it with the Rack configuration <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">config.ru</code> file.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'./demo_api_classic'</span>
run <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">config.ru</code> file in place, the application is started with the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">rackup config.ru
</code></pre>
<h3 id="modular-approach" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Modular Approach</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Modular Sinatra applications are created by subclassing either <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sinatra::Base</code> or <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sinatra::Application</code>:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'sinatra'</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DemoApi</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span></span></span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># Application code</span>
run! <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> app_file == <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$0</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The statement beginning with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">run!</code> is used for starting the application directly, with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ruby demo_api.rb</code>, just as with the classical application. On the other hand, if the application is to be deployed with Rack, the handlers content of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">rackup.ru</code> must be:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'./demo_api'</span>
run <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DemoApi</span>
</code></pre>
<h2 id="sequel" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sequel</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sequel is the second tool in this set. In contrast to ActiveRecord, which is part of the Ruby on Rails, Sequel’s dependencies are very small. At the same time, it is quite feature rich and can be used for all kinds of database manipulation tasks. With its simple domain specific language, Sequel relieves the developer from all the problems with maintaining connections, constructing SQL queries, fetching data from (and sending data back to) the database.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For example, establishing a connection with the database is very simple:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DB</span> = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel</span>.connect(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">adapter:</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:postgres</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">database:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'my_db'</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">host:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'localhost'</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">user:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'db_user'</span>)
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The connect method returns a database object, in this case, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sequel::Postgres::Database</code>, which can be further used to execute raw SQL.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DB</span>[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'select count(*) from players'</span>]
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Alternatively, to create a new dataset object:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DB</span>[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:players</span>]
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Both of these statements create a dataset object, which is a basic Sequel entity.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One of the most important Sequel dataset features is that it does not execute queries immediately. This makes it possible to store datasets for later use and, in most cases, to chain them.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">users = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DB</span>[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:players</span>].where(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">sport:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'tennis'</span>)
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, if a dataset does not hit the database immediately, the question is, when does it? Sequel executes SQL on the database when so-called “executable methods” are used. These methods are, to name a few, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">all</code>, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">each</code>,<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">map</code>, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">first</code>, and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">last</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sequel is extensible, and its extensibility is a result of a fundamental architectural decision to build a small core complemented with a plugin system. Features are easily added through plugins which are, actually, Ruby modules. The most important plugin is the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Model</code> plugin. It is an empty plugin which does not define any class or instance methods by itself. Instead, it includes other plugins (submodules) which define a class, instance or model dataset methods. The Model plugin enables the use of Sequel as the object-relational-mapping (ORM) tool and is often referred to as the “base plugin”.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Player</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel::Model</span></span></span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Sequel model automatically parses the database schema and sets up all necessary accessor methods for all columns. It assumes that table name is plural and is an underscored version of the model name. In case there is a need to work with databases that do not follow this naming convention, the table name can be explicitly set when the model is defined.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">class Player < Sequel::Model(:player)
end
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, we now have everything we need to start building the back-end API.</div>
<h2 id="building-the-api" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Building the API</h2>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-email_form is-current" style="border: 0px; box-sizing: border-box; display: flex; flex-wrap: wrap; height: auto; margin: 0px; min-height: 0px; min-width: 0px; overflow: visible; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<div class="embeddable_form-label_title" style="border: 0px; box-sizing: border-box; color: #2557a1; display: inline-block; font-size: 17px; font-weight: 700; line-height: 23px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Like what you're reading?</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Get the latest updates first.</div>
</div>
<div class="embeddable_form-row form-field is-email_field" style="border: 0px; box-sizing: border-box; flex-basis: 100%; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<input autocomplete="off" class="input is-medium" data-role="email" name="blog_subscription[email]" placeholder="Enter your email address..." style="-webkit-appearance: none; background: rgb(250, 250, 250); border-color: rgb(238, 238, 238); border-radius: 4px; border-style: solid; border-width: 1px; color: #3c3c3c; font-family: proxima-nova, Arial, sans-serif; font-size: 14px; margin-bottom: 0px; margin-top: 0px; padding: 15px 12px; transition: all 0.2s; width: 864px;" type="text" /></div>
<div class="embeddable_form-row is-submit" style="-webkit-box-flex: 1; -webkit-box-ordinal-group: 4; border: 0px; box-sizing: border-box; flex-grow: 1; margin: 0px; min-height: 0px; min-width: 0px; order: 3; padding: 0px; position: relative; vertical-align: baseline;">
<input class="button is-green_candy is-default is-full_width" data-loader-text="Subscribing..." data-role="submit" style="-webkit-appearance: none; background-attachment: initial; background-clip: initial; background-image: linear-gradient(rgb(67, 198, 146), rgb(57, 184, 133)); background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-color: rgb(31, 124, 87); border-radius: 4px; border-style: solid; border-width: 1px; box-shadow: rgb(79, 211, 170) 0px 1px inset; color: white; cursor: pointer; font-size: 14px; font-weight: 600; padding: 15px 20px; position: relative; text-shadow: rgb(28, 143, 61) 0px 1px 0px; transition: background 150ms; width: 552.672px;" type="submit" value="Get Exclusive Updates" /></div>
<div class="embeddable_form-row is-privacy" style="border: 0px; box-sizing: border-box; margin: 0px 20px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<div class="embeddable_form-privacy" style="-webkit-box-align: center; align-items: center; border: 0px; box-sizing: border-box; color: #9a9a9a; display: flex; font-size: 12px; line-height: 50px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-privacy_icon" style="background: url("/assets/front/static/public/primitives/icon/lock_ed584f.png") 0% 0% / 9.5px 11.5px no-repeat; border: 0px; box-sizing: border-box; display: inline-block; flex-shrink: 0; height: 11.5px; margin: 0px 15px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; top: -2px; vertical-align: baseline; width: 9.5px;">
</div>
<div class="embeddable_form-privacy_text" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
No spam. Just great engineering and design posts.</div>
</div>
</div>
</div>
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label_title" style="border: 0px; box-sizing: border-box; color: #2557a1; display: inline-block; font-size: 17px; font-weight: 700; line-height: 23px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label is-header" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a data-role="preferences_link" href="https://www.toptal.com/ruby/api-with-sinatra-and-sequel-ruby-tutorial#" style="border: 0px; box-sizing: border-box; color: #3976cb; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;"></a></div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptal" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-image: initial; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-right-style: initial; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-image: initial; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-right-color: initial; border-right-style: initial; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-image: initial; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-right-color: initial; border-right-style: initial; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="code-structure" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Code Structure</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Contrary to Rails, Sinatra does not impose any project structure. However, since it is always a good practice to organize code for easier maintenance and development, we’ll do it here too, with the following directory structure:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">project root
|-config
|-helpers
|-models
|-routes
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The application configuration will be loaded from the YAML configuration file for the current environment with:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span>.config_file <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.join(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.dirname(__FILE_<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">_</span>),
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'config'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span>.settings.environment}</span>_config.yml"</span>)
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By default, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sinatra::Applicationsettings.environment</code> value is <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">development,</code> and it is changed by setting the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">RACK_ENV</code> environment variable.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Furthermore, our application must load all files from the other three directories. We can do that easily by running:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">%w{helpers models routes}</span>.each {|dir| <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Dir</span>.glob(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{dir}</span>/*.rb"</span>, &method(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:require</span>))}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
At first glance, this way of loading might look convenient. However, with this one line of the code, we cannot easily skip files, because it will load all the files from the directories in the array. That’s why we will use a more efficient single-file load approach, which assumes that in each folder we have a manifest file <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">init.rb</code>, which loads all other files from the directory. Also, we will add a target directory to the Ruby load path:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">%w{helpers models routes}</span>.each <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |dir|
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">$LOAD_PATH</span> << <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.expand_path(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'.'</span>, <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.join(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.dirname(__FILE_<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">_</span>), dir))
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.join(dir, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'init'</span>)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This approach requires a bit more work because we have to maintain require statements in each <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">init.rb</code> file but, in return, we get more control, and we can easily leave one or more files out by removing them from the manifest <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">init.rb</code> file in the target directory.</div>
<h3 id="api-authentication" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
API Authentication</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first thing we need in each API is authentication. We will implement it as a helper module. Complete authentication logic will be in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">helpers/authentication.rb</code> file.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'multi_json'</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">module</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra</span></span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">module</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Authentication</span></span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>authenticate!
client_id = request[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'client_id'</span>]
client_secret = request[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'client_secret'</span>]
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># Authenticate client here</span>
halt <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">401</span>, <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump({<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">message:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"You are not authorized to access this resource"</span>}) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">unless</span> authenticated?
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>current_client
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@current_client</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>authenticated?
!current_client.<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">nil</span>?
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
helpers <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Authentication</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All we have to do now is to load this file by adding a require statement in the helper manifest file (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">helpers/init.rb</code>) and to call the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">authenticate!</code> method in Sinatra’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">before</code> hook which will be executed before processing any request.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">before <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
authenticate!
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<h3 id="database" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Database</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Next, we have to prepare our database for the application. There are many ways to prepare the database, but since we are using Sequel, it is natural to do it by using migrators. Sequel comes with two migrator types - integer and timestamp based. Each one has its advantages and drawbacks. In our example, we decided to use the Sequel’s timestamp migrator, which requires migration files to be prefixed with a timestamp. The timestamp migrator is very flexible and can accept various timestamp formats, but we will only use the one that consists of year, month, day, hour, minute, and second. Here are our two migration files:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># db/migrations/20160710094000_sports.rb</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel</span>.migration <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
change <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
create_table(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:sports</span>) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
primary_key <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">String</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:name</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:null</span> => <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># db/migrations/20160710094100_players.rb</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel</span>.migration <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
change <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
create_table(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:players</span>) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
primary_key <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">String</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:name</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:null</span> => <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>
foreign_key <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:sport_id</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:sports</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We are now ready to create a database with all the tables.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bundle exec sequel -m db/migrations <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">sqlite:</span>/<span class="hljs-regexp" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/db/development</span>.sqlite3
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Finally, we have the model files <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sport.rb</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">player.rb</code> in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">models</code> directory.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># models/sport.rb</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sport</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel::Model</span></span></span>
one_to_many <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:players</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>to_api
{
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">id:</span> id,
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name:</span> name
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># models/player.rb</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Player</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sequel::Model</span></span></span>
many_to_one <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:sport</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>to_api
{
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">id:</span> id,
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name:</span> name,
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">sport_id:</span> sport_id
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here we are employing a Sequel way of defining model relationships, where the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sport</code> object has many players and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Player</code> can have only one sport. Also, each model defines its <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">to_api</code> method, which returns a hash with attributes that need to be serialized. This is a general approach we can use for various formats. However, if we will only use a JSON format in our API, we could use Ruby’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">to_json</code> with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">only</code> argument to restrict serialization to required attributes, i.e. <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">player.to_json(only: [:id, :name, :sport_i])</code>. Of course, we could also define a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">BaseModel</code> that inherits from <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Sequel::Model</code> and defines a default <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">to_api</code> method, from which inherit all models could then inherit.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, we can start implementing the actual API endpoints.</div>
<h3 id="api-endpoints" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
API Endpoints</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We will keep the definition of all endpoints in files within the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">routes</code> directory. Since we are using manifest files for loading files, we will group routes by resources (i.e., keep all sports related routes in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sports.rb</code> file, all players routes in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">routes.rb</code>, and so on).</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># routes/sports.rb</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DemoApi</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span></span></span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/sports/?"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sport</span>.all.map { |s| s.to_api })
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/sports/:id"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
sport = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sport</span>.where(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">id:</span> params[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>]).first
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump(sport ? sport.to_api <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:</span> {})
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/sports/:id/players/?"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
sport = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sport</span>.where(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">id:</span> params[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>]).first
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump(sport ? sport.players.map { |p| p.to_api } <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:</span> [])
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># routes/players.rb</span>
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DemoApi</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Sinatra::Application</span></span></span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/players/?"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Player</span>.all.map { |p| s.to_api })
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/players/:id/?"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
player = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Player</span>.where(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">id:</span> params[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>]).first
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiJson</span>.dump(player ? player.to_api <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:</span> {})
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Nested routes, like the one for getting all the players within one sport <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">/sports/:id/players</code>, can either be defined by placing them together with other routes, or by creating a separate resource file that will contain only the nested routes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With designated routes, the application is now ready to accept requests:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">curl -i -XGET 'http://localhost:9292/sports?client_id=<client_id>&client_secret=<client_secret>'
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that as required by the application’s authentication system defined in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">helpers/authentication.rb</code> file, we are passing credentials directly in request parameters.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #3863a0; font-size: 1.3em; text-align: center;">Conclusion</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The principles demonstrated in this simple example application apply to any API back-end application. It is not based on the model-view-controller (MVC) architecture, yet it keeps a clear separation of responsibilities in a similar way; complete business logic is kept in model files while handling requests is done in Sinatra’s routes methods. Contrary to MVC architecture, where views are used to render responses, this application does that at the same place where it handles requests - in the routes methods. With new helper files, the application can be easily extended to send pagination or, if needed, request limits information back to the user in response headers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the end, we’ve built a complete API with a very simple toolset and without losing any functionality. The limited number of dependencies helps ensure that the application loads and starts much faster, and has a much smaller memory footprint than a Rails based one would have. So, next time you start work on a new API in <a href="https://www.toptal.com/ruby" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ruby</a>, consider using Sinatra and Sequel since they are very powerful tools for such a use case.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/bosko-ivanisevic" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Boško Ivanišević</a>, a <a href="https://www.toptal.com/ruby/api-with-sinatra-and-sequel-ruby-tutorial">Toptal</a><a href="https://www.toptal.com/technical-writers"> freelance developer</a>.</div>
Anonymoushttp://www.blogger.com/profile/14742130202178750860noreply@blogger.com5tag:blogger.com,1999:blog-8706416286757464743.post-87897120910633208302016-09-09T12:43:00.001+07:002016-09-09T12:43:49.494+07:00The Six Commandments of Good Code: Write Code that Stands the Test of Time<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Humans have only been grappling with the art and science of computer programming for roughly half a century. Compared to most arts and sciences, computer science is in many ways still just a toddler, walking into walls, tripping over its own feet, and occasionally throwing food across the table. As a consequence of its relative youth, I don’t believe we have a consensus yet on what a proper definition of “good code” is, as that definition continues to evolve. Some will say “good code” is code with 100% test coverage. Others will say it’s super fast and has a killer performance and will run acceptably on 10 year old hardware. While these are all laudable goals for software developers, however I venture to throw another target into the mix: maintainability. Specifically, “good code” is code that is easily and readily maintainable by an organization (not just by its author!) and will live for longer than just the sprint it was written in. The following are some things I’ve discovered in my career as an engineer at big companies and small, in the USA and abroad, that seem to correlate with maintainable, “good” software.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/121126/toptal-blog-image-1472743313571-54196255310a8830c39c00274bc0d780.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Never settle for code that just "works." Write superior code.</span></div>
<h2 id="commandment-1-treat-your-code-the-way-you-want-others-code-to-treat-you" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #1: Treat Your Code the Way You Want Other’s Code to Treat You</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I’m far from the first person to write that the primary audience for your code is not the compiler/computer, but whomever next has to read, understand, maintain, and enhance the code (which will not necessarily be you 6 months from now). Any engineer worth their pay can produce code that “works”; what distinguishes a superb engineer is that they can write maintainable code efficiently that supports a business long term, and have the skill to solve problems simply and in a clear and maintainable way.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In any programming language, it is possible to write good code or bad code. Assuming we judge a programming language by how well it facilitates writing good code (it should at least be one of the top criteria, anyway), any programming language can be “good” or “bad” depending on how it is used (or abused).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
An example of a language that by many is considered ‘clean’ and readable is Python. The language itself enforces some level of white space discipline and the built in APIs are plentiful and fairly consistent. That said, it’s possible to create unspeakable monsters. For example, one can define a class and define/redefine/undefine any and every method on that class during runtime (often referred to as monkey patching). This technique naturally leads to at best an inconsistent API and at worst an impossible to debug monster. One might naively think,”sure, but nobody does that!” Unfortunately they do, and it doesn’t take long browsing <a href="http://pypi.python.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pypi</a> before you run into substantial (and popular!) libraries that (ab)use monkey patching extensively as the core of their APIs. I recently used a networking library whose entire API changes depending on the network state of an object. Imagine, for example, calling <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">client.connect()</code> and sometimes getting a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">MethodDoesNotExist</code> error instead of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">HostNotFound</code> or <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">NetworkUnavailable</code>.</div>
<h2 id="commandment-2-good-code-is-easily-read-and-understood-in-part-and-in-whole" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #2: Good Code Is Easily Read and Understood, in Part and in Whole</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Good code is easily read and understood, in part and in whole, by others (as well as by the author in the future, trying to avoid the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">“Did I really write that?”</em> syndrome).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By “in part” I mean that, if I open up some module or function in the code, I should be able to understand what it does without having to also read the entire rest of the codebase. It should be as intuitive and self-documenting as possible.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Code that constantly references minute details that affect behavior from other (seemingly irrelevant) portions of the codebase is like reading a book where you have to reference the footnotes or an appendix at the end of every sentence. You’d never get through the first page!</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Some other thoughts on “local” readability:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Well <a href="https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">encapsulated</a> code tends to be more readable, <a href="https://en.wikipedia.org/wiki/Separation_of_concerns" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">separating concerns</a> at every level.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Names matter.</span> Activate <a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Thinking Fast and Slow’s</a><a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow#Two_systems" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">system 2</a> way in which the brain forms thoughts and put some actual, careful thought into variable and method names. The few extra seconds can pay significant dividends. A well-named variable can make the code much more intuitive, whereas a poorly-named variable can lead to headfakes and confusion.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Cleverness is the enemy.</span> When using fancy techniques, paradigms, or operations (such as list comprehensions or ternary operators), be careful to use them in a way that makes your code <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">more </span>readable, not just shorter.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Consistency is a good thing.</span> Consistency in style, both in terms of how you place braces but also in terms of operations, improves readability greatly.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://en.wikipedia.org/wiki/Separation_of_concerns" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Separation of concerns</a>.</span> A given project manages an innumerable number of locally important assumptions at various points in the codebase. Expose each part of the codebase to as few of those concerns as possible. Say you had a people management system where a person object may sometimes have a null last name. To somebody writing code in a page that displays person objects, that could be really awkward! And unless you maintain a handbook of “Awkward and non obvious assumptions our codebase has” (I know I don’t) your display page programmer is not going to know last names can be null and is probably going to write code with a null pointer exception in it when the last name-being null case shows up. Instead handle these cases with well thought out APIs and contracts that different pieces of your codebase use to interact with each other.</div>
</li>
</ul>
<h2 id="commandment-3-good-code-has-a-well-thought-out-layout-and-architecture-to-make-managing-state-obvious" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #3: Good Code Has a Well Thought-out Layout and Architecture to Make Managing State Obvious</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">State is the enemy.</em> Why? Because it is the single most complex part of any application and needs to be dealt with very deliberately and thoughtfully. Common problems include database inconsistencies, partial UI updates where new data isn’t reflected everywhere, out of order operations, or just mind numbingly complex code with if statements and branches everywhere leading to difficult to read and even harder to maintain code. Putting state on a pedestal to be treated with great care, and being extremely consistent and deliberate with regard to how state is accessed and modified, dramatically simplifies your codebase. Some languages (Haskell for example) enforce this at a programmatic and syntactic level. You’d be amazed how much the clarity of your codebase can improve if you have libraries of <a href="https://en.wikipedia.org/wiki/Pure_function" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pure</a> functions that access no external state, and then a small surface area of stateful code which references the outside pure functionality.</div>
<h2 id="commandment-4-good-code-doesnt-reinvent-the-wheel-it-stands-on-the-shoulders-of-giants" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #4: Good Code Doesn’t Reinvent the Wheel, it Stands on the Shoulders of Giants</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Before potentially reinventing a wheel, think about how common the problem is you’re trying to solve or the function is you’re trying to perform. Somebody may have already implemented a solution you can leverage. Take the time to think about and research any such options, if appropriate and available.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/121144/toptal-blog-image-1473062300568-7ece4318b08e18915dea58450d3cb7f7.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
That said, a completely reasonable counter-argument is that dependencies don’t come for “free” without any downside. By using a 3rd party or open source library that adds some interesting functionality, you are making the commitment to, and becoming dependent upon, that library. That’s a big commitment; if it’s a giant library and you only need a small bit of functionality do you really want the burden of updating the whole library if you upgrade, for example, to Python 3.x? And moreover, if you encounter a bug or want to enhance the functionality, you’re either dependent on the author (or vendor) to supply the fix or enhancement, or, if it’s open source, find yourself in the position of exploring a (potentially substantial) codebase you’re completely unfamiliar with trying to fix or modify an obscure bit of functionality.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Certainly the more well used the code you’re dependent upon is, the less likely you’ll have to invest time yourself into maintenance. The bottom line is that it’s worthwhile for you to do your own research and make your own evaluation of whether or not to include outside technology and how much maintenance that particular technology will add to your stack.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Below are some of the more common examples of things you should probably not be reinventing in the modern age in your project (unless these ARE your projects).</div>
<h3 id="databases" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Databases</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Figure out which of <a href="http://en.wikipedia.org/wiki/CAP_theorem" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">CAP</a> you need for your project, then chose the database with the right properties. Database doesn’t just mean MySQL anymore, you can chose from:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">“Traditional” Schema’ed SQL:</span> Postgres / MySQL / MariaDB / MemSQL / Amazon RDS, etc.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Key Value Stores:</span> Redis / Memcache / Riak</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">NoSQL:</span> MongoDB/Cassandra</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Hosted DBs:</span> AWS RDS / DynamoDB / AppEngine Datastore</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Heavy lifting:</span> Amazon MR / Hadoop (Hive/Pig) / Cloudera / Google Big Query</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Crazy stuff:</span> Erlang’s Mnesia, iOS’s Core Data</li>
</ul>
<h3 id="data-abstraction-layers" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Data Abstraction Layers</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should, in most circumstances, not be writing raw queries to whatever database you happen to chose to use. More likely than not, there exists a library to sit in between the DB and your application code, separating the concerns of managing concurrent database sessions and details of the schema from your main code. At the very least, you should never have raw queries or SQL inline in the middle of your application code. Rather, wrap it in a function and centralize all the functions in a file called something really obvious (e.g., “queries.py”). A line like <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">users = load_users()</code>, for example, is infinitely easier to read than <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">users = db.query(“SELECT username, foo, bar from users LIMIT 10 ORDER BY ID”)</code>. This type of centralization also makes it much easier to have consistent style in your queries, and limits the number of places to go to change the queries should the schema change.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptal" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="other-common-libraries-and-tools-to-consider-leveraging" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Other Common Libraries and Tools to Consider Leveraging</h3>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Queuing or Pub/Sub Services.</span> Take your pick of AMQP providers, ZeroMQ, RabbitMQ, Amazon SQS</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Storage.</span> Amazon S3, Google Cloud Storage</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Monitoring:</span> Graphite/<a href="https://www.hostedgraphite.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Hosted Graphite</a>, AWS Cloud Watch, <a href="https://newrelic.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">New Relic</a></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Log Collection / Aggregation.</span> <a href="https://www.loggly.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Loggly</a>, <a href="http://www.splunk.com/en_us/products/splunk-cloud.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Splunk</a></li>
</ul>
<h3 id="auto-scaling" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Auto Scaling</h3>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Auto Scaling.</span> Heroku, AWS Beanstalk, AppEngine, AWS Opsworks, <a href="https://www.digitalocean.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Digital Ocean</a></li>
</ul>
<h2 id="commandment-5-dont-cross-the-streams" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #5: Don’t Cross the Streams!</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are many good models for programming <a href="https://www.toptal.com/designers/web/portfolios">design</a>, <a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pub/sub</a>, <a href="https://en.wikipedia.org/wiki/Actor_model" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">actors</a>, <a href="https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">MVC</a> etc. Choose whichever you like best, and stick to it. Different kinds of logic dealing with different kinds of data should be physically isolated in the codebase (again, this separation of concerns concept and reducing cognitive load on the future-reader). The code which updates your UI should be physically distinct from the code that calculates what goes into the UI, for example.</div>
<h2 id="commandment-6-when-possible-let-the-computer-do-the-work" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Commandment #6: When Possible, Let the Computer Do the Work</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If the compiler can catch logical errors in your code and prevent either bad behavior, bugs, or outright crashes, we absolutely should take advantage of that. Of course, some languages have compilers that make this easier than others. Haskell, for example, has a famously strict <a href="https://en.wikipedia.org/wiki/Glasgow_Haskell_Compiler" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">compiler</a> that results in programmers spending most of their effort just getting code to compile. Once it compiles though, “it just works”. For those of you who’ve either never written in a strongly typed functional language this may seem ridiculous or impossible, but <a href="https://www.quora.com/Why-is-that-some-people-say-that-if-a-Haskell-code-compiles-it-probably-works" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">don’t</a> <a href="https://wiki.haskell.org/Why_Haskell_just_works" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">take</a> <a href="https://news.ycombinator.com/item?id=8392945" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">my</a> <a href="http://www.drdobbs.com/architecture-and-design/in-praise-of-haskell/240163246" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">word</a> for it. Seriously, click on some of these links, it’s absolutely possible to live in a world without runtime errors. And it really is that magical.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Admittedly, not every language has a compiler or a syntax that lends itself to much (or in some cases any!) compile-time checking. For those that don’t, take a few minutes to research what optional strictness checks you can enable in your project and evaluate if they make sense for you. A short, non-comprehensive list of some common ones I’ve used lately for languages with lenient runtimes include:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Python: <a href="https://www.pylint.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pylint</a>, <a href="https://pypi.python.org/pypi/pyflakes" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pyflakes</a>, <a href="https://docs.python.org/2/library/warnings.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">warnings</a>, <a href="https://www.emacswiki.org/emacs/PythonProgrammingInEmacs" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">warnings in emacs</a></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ruby: <a href="http://devblog.avdi.org/2011/06/23/how-ruby-helps-you-fix-your-broken-code/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">warnings</a></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Javascript: <a href="http://www.jslint.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">jslint</a></li>
</ul>
<h2 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Conclusion</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is by no means an exhaustive or the perfect list of commandments for producing “good” (i.e., easily maintainable) code. That said, if every codebase I ever had to pick up in the future followed even half of the concepts in this list, I will have many fewer gray hairs and might even be able to add an extra 5 years on the end of my life. And I’ll certainly find work more enjoyable and less stressful.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/zachary-goldberg" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Zachary Goldberg</a>, a <a href="https://www.toptal.com/software/six-commandments-of-good-code">Toptal</a><a href="https://www.toptal.com/technical-writers"> freelance developer</a>.</div>
Anonymoushttp://www.blogger.com/profile/14742130202178750860noreply@blogger.com5tag:blogger.com,1999:blog-8706416286757464743.post-60074114296596929092016-08-17T18:21:00.002+07:002016-08-17T18:21:25.518+07:00Developing Mobile Web Apps: When, Why, and How<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are 6.8 billion people on the planet, 5.1 billion of whom own a cell phone (<a href="http://blog.hubspot.com/blog/tabid/6307/bid/24082/9-Amazing-Mobile-Marketing-Statistics-Every-Marketer-Should-Know.aspx" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">source</a>). And today, an ever-growing percentage of these devices are smartphones. According to a recent <a href="http://www.pewinternet.org/files/old-media//Files/Reports/2013/PIP_Cell%20Phone%20Activities%20May%202013.pdf" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Pew Research Center Study</a>, the number of users accessing the Internet on their smartphones has more than doubled in the past 5 years, as has the number of users downloading and using <a href="https://www.toptal.com/designers/mobile/portfolios" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">mobile apps</a>. Of those who use the Internet or email on their phones, more than a third go online primarily through their handheld devices.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Indeed, mobile computing is becoming increasingly ubiquitous… and it’s awesome.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Except, of course, when it’s not.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As a mobile device user, few things are as frustrating and difficult to fat-finger-navigate as a poorly designed mobile web or native app.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And as a <a href="https://www.toptal.com/android" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">mobile app developer</a>, few things can be as intensely irritating as striving to support as wide a range of mobile clients as possible, each of which has its own frustrating set of idiosyncrasies. Whether you choose to develop a mobile web, native, or hybrid app, the quest to support <a href="http://lifehacker.com/5925969/five-best-android-web-browsers" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">multiple mobile browsers</a>, <a href="https://www.toptal.com/google-glass/building-your-first-glass-app-a-tutorial-for-aspiring-google-glass-developers" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">more-exotic devices</a>, and platforms can be quite a gut wrenching experience indeed.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="This web app development tutorial seeks to help you navigate different browsers and platforms." src="https://assets.toptal.io/uploads/blog/image/315/toptal-blog-image-1394581973091.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; font-style: italic; line-height: 1.5em;">As a mobile device user, few things are as frustrating and difficult to fat-finger-navigate as a poorly designed mobile web or native app. And as a mobile app developer, few things can be as intensely irritating as striving to support as wide a range of mobile clients as possible, each of which has its own frustrating set of idiosyncrasies.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Of course, not every developer today needs to worry about supporting mobile clients. But the increasingly omnipresent nature of mobile devices and applications strongly suggests that those who don’t need to support mobile clients today will more than likely need to do so in the not-too-distant future. So if you’re not already thinking about mobile app development, you probably should be.</div>
<h2 id="mobile-app--web-vs-native-vs-hybrid-help-me-choose" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Mobile app: Web vs. native vs. hybrid (help me choose!)</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As is true with most technology selections, there’s no one-size-fits-all answer when it comes to the type of mobile app to develop. There are numerous web app best practices to consider, not all of which are technical. Who is your target audience? Are they more likely to prefer a mobile web or a native app? What <a href="https://www.toptal.com/resources" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">development resources</a> do you have and which mobile technologies are they most familiar with? What is the licensing and sales model that you’re envisioning for your product?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Generally speaking (although there are always exceptions), the mobile web route is faster and cheaper than the native app route, especially when the objective is to support a wide range of devices. Conversely, there may be capabilities native to the mobile device (such as the movement sensor and so on) that are essential to your app, but which are only accessible via a native app (which would therefore make the mobile web app choice a non-starter for you).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And beyond the web vs. native question, a hybrid app may be the right answer for you, depending on your requirements and resource constraints. Hybrid apps, like native apps, run on the device itself (as opposed to inside a browser), but are written with web technologies (HTML5, CSS and JavaScript). More specifically, hybrid apps run inside a native container, and leverage the device’s browser engine (but not the browser) to render the HTML and process the JavaScript locally. A web-to-native abstraction layer enables access to device capabilities that are not accessible in mobile web applications, such as the accelerometer, camera, and local storage.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But whatever choice you make – whether it be mobile web, native or hybrid app – be careful to adequately research and confirm your assumptions. As an example for the purposes of this mobile web app development tutorial, you may have decided to develop a native app for e-commerce to sell your products, but according to<a href="http://www.slideshare.net/HubSpot/50-mobilefactsdeck62812" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Hubspot</a>, 73% of smartphone users say they use the mobile web more than native apps to do their shopping… so you may have bet on the wrong horse.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.3em; font-style: italic; line-height: 1.5em;">But whatever choice you make – whether it be mobile web, native or hybrid app – be careful to adequately research and confirm your assumptions.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And then, of course, there are the practical considerations of time and budget. As one of my favorite sayings goes, “faster, better, cheaper… pick any two”. While time-to-market and cost constraints are of paramount importance in web application development, it’s crucial not to compromise too heavily on quality in the process. It’s quite difficult to recover the confidence of a user who has had a bad first experience.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Indeed, mobile web, native, and hybrid apps are all radically different beasts, each with their own unique set of benefits and challenges. This development tutorial specifically focuses on methodologies and tools to employ, and pitfalls to avoid, in the development of highly functional, intuitive, and easy-to-use mobile web applications.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="A critical best practice in determining how to develop a mobile web application is to know your customer." src="https://assets.toptal.io/uploads/blog/image/311/toptal-blog-image-1394506935807.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="plan-ahead-if-you-dont-know-where-youre-going-you-just-might-end-up-there" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Plan ahead (“if you don’t know where you’re going, you just might end up there…”)</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Identifying your (or your customer’s) requirements is one of the most essential best practices in app development, mobile or otherwise. Carefully research the targeted capabilities to determine if they are achievable in a web app. It’s quite frustrating, and highly unproductive, to realize that one or more of your essential client functions aren’t supported, when you’ve already invested the time and resources to design the web-based interface and supporting infrastructure.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another common gotcha for mobile web app developer newbies is to ass-u-me that web-based code for a desktop browser will work “as is” in a mobile browser. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Not.</em> There most definitely <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">are</em> differences and, if you’re not aware of them, they can definitely bite you. The HTML5 <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><video></code> tag’s autoplay functionality, for example, doesn’t work on mobile browsers. Similarly, the CSS <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">transition</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">opacity</code> properties are not supported (or at least are not consistently supported) in most mobile browsers nowadays. You will also have problems with some web API methods on a mobile platform, such as the SoundCloud music streaming API that requires Adobe Flash which is not supported on most mobile devices.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.3em; font-style: italic; line-height: 1.5em;">A common gotcha for mobile web app developer newbies is to ass-u-me that web-based code for a desktop browser will work “as is” in a mobile browser.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A particularly complicating factor in mobile web application development is that the lifespan of mobile devices tends to be much shorter than that of desktop displays (the average lifespan of a cell phone in the U.S. is around <a href="http://smallbusiness.chron.com/life-expectancy-smartphone-62979.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">21 months</a>). These shorter device life spans, accompanied by constant releases of new mobile devices and technologies, yield an ever-changing landscape of to-be-targeted devices. While working in a browser does somewhat alleviate this issue by shielding you from a number of device-specific issues, you will still need to design a browser-based view that supports many different screen resolutions (as well as adjusting appropriately for landscape and portrait orientations).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Thought needs to be given as well to supporting Apple’s Retina Displays (liquid crystal displays that have a pixel density high enough that the human eye is unable to discern individual pixels at a typical viewing distance). Several Apple products – including the <a href="https://www.toptal.com/designers/iphone/portfolios" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">iPhone</a>, iPod Touch, iPad, MacBook Pro, iPad Mini, and iPad Air – offer Retina displays. For a mobile web app in particular, it’s important to be aware that a Retina display makes low resolution images (which are typically served to mobile devices) look fuzzy and <a href="https://en.wikipedia.org/wiki/Pixelation" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pixelation</a> can occur. The best app development solution in these cases is to have the server recognize that the request is coming from a Retina device and to then provide an alternate higher resolution image to the client.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you want to use some of the cool <a href="https://www.toptal.com/html5/job-description" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">HTML5 stuff</a>, remember to verify in advance that the functionality you’re looking for is supported across the device landscape that your customers are likely to be using. For example, in iOS 6 and above, there is no support for the navigator <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">getUserMedia</code> functionality since the camera is only accessible through native apps. Two great resources for checking what’s supported on specific devices and browsers are <a href="http://caniuse.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">caniuse.com</a> and <a href="http://html5test.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">html5test.com</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.3em; font-style: italic; line-height: 1.5em;">Remember to verify in advance that the functionality you’re looking for is supported across the device landscape that your customers are likely to be using.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
CSS3 media queries can also help you provide customized content for each device. Here’s some example code for capturing different device characteristics, such as pixel density, screen resolution, and orientation:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">/* For lower than 700px resolutions */
@media (max-width: 700px) { ... }
/* Same as last but with the device orientation on land scape */
@media (max-width: 700px) and (orientation: landscape) { ... }
/* Including width and orientation you can add a media type clause,
in this case 'tv' */
@media tv and (min-width: 700px) and (orientation: landscape) { ... }
/* for low resolution display with background-image */
.image {
background-image: url(/path/to/my/image.png);
background-size: 200px 300px;
height: 300px;
width: 200px;
}
/* for high resolution (Retina) display with background-image */
@media only screen and (min--moz-device-pixel-ratio: 2),
only screen and (-o-min-device-pixel-ratio: 2/1),
only screen and (-webkit-min-device-pixel-ratio: 2),
only screen and (min-device-pixel-ratio: 2) {
-repeat;
background-size: 200px 400px;
/* rest of your styles... */
}
}
</code></pre>
<h2 id="performance-performance-performance" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Performance, performance, performance</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">“OMG, this thing is sooooo slow!”</em> As a mobile web app developer, those are probably the very last words you ever want to hear from one of your users. You must therefore think carefully about how to reduce and optimize each byte and server transfer to reduce the user’s wait time. It’s unrealistic to expect that transfers will always be done over a WiFi network, and you should know that <a href="http://go-n-mobile.com/docs/50facts.pdf" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">60% of mobile web users</a> say they expect a site to load on their mobile phone in 3 seconds or less (source). Similarly, Google found that, for every extra 5 seconds of load time, traffic dropped by 20% (and it is also worth noting that search engines look at load times as part of their calculation of page quality score).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.3em; font-style: italic; line-height: 1.5em;">60% of mobile web users say they expect a site to load on their mobile phone in 3 seconds or less.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As a part of this web app development tutorial, here are a few tips that can help optimize the performance of your mobile web app and minimize latency:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Image Optimization.</em></span> Image load time is well-known to be one of the biggest performance issues affecting page load on mobile devices. Use of online image optimizers, such as <a href="http://www.smushit.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">smushit.com</a>, can be helpful in addressing this issue.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Code compression.</em></span> Compressing your JavaScript and CSS files, depending on the amount of code you have, can potentially have a significant impact on performance. A useful tool for compressing your code is<a href="http://www.refresh-sf.com/yui/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">refresh-sh.com</a>.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Database queries.</em></span><ul style="border: 0px; box-sizing: border-box; font-size: 1em; list-style: none; margin: 1em 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: circle; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Some mobile device browsers don’t accept as many cookies as desktop browsers do, which can result in the need to execute even more queries than usual. Server-side caching is therefore especially crucial when supporting mobile web app clients.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: circle; margin: 0px 0px 0px 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Remember to employ the appropriate filters to preclude SQL query injection that could otherwise compromise the security of your site and server.</li>
</ul>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Content delivery networks (CDN).</em></span> If you are planning to provide lots of videos, images, audio files, or other types of media, use of a CDN is highly recommended. Some of the more common commercial CDNs include <a href="http://aws.amazon.com/s3/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Amazon S3</a>, <a href="https://www.windowsazure.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Microsoft Windows Azure</a>, and <a href="http://www.maxcdn.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">MaxCDN</a>. The advantages of using a CDN are numerous and include:<ul style="border: 0px; box-sizing: border-box; font-size: 1em; list-style: none; margin: 1em 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: circle; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Improved download performance.</em> Leveraging a CDN’s resources enables you to distribute load, save bandwidth, and boost performance. The better CDNs offer higher availability, lower network latency, and lower packet loss. Moreover, many CDNs provide a globally distributed selection of data centers, enabling downloads to occur from a server closer to the user’s location (resulting in fewer network hops and faster downloads).</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: circle; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">More concurrent downloads.</em> Browsers typically limit the number of concurrent connections to a single domain, after which additional downloads are blocked until one of the previous downloads has completed. You can often see this limit in action when downloading many large files from the same site. Each additional CDN (on a different domain) allows for additional concurrent downloads.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: circle; margin: 0px 0px 0px 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Enhanced analytics.</em> Many commercial CDNs provide usage reports that can supplement your own website analytics and which may offer a better quantification of video views and downloads. <a href="http://gtmetrix.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">GTmetrix</a>, for example, has an excellent website reporting tool for monitoring and optimizing the sources loaded on your site.</li>
</ul>
</li>
</ul>
<h2 id="your-mobile-web-app-development-toolbox" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Your mobile web app development toolbox</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
“The right tools for the right job” is an age-old adage that applies as much to software development as it does to any other domain. This tutorial provides and introduction to some of the more popular and widely-used tools for mobile web app development, but bear in mind that there may very well be other tools that are the “right” ones for developing your mobile web app, depending on your requirements and available resources.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label_title" style="border: 0px; box-sizing: border-box; color: #2557a1; display: inline-block; font-size: 17px; font-weight: 700; line-height: 23px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label is-header" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a data-role="preferences_link" href="https://www.toptal.com/android/developing-mobile-web-apps-when-why-and-how#" style="border: 0px; box-sizing: border-box; color: #3976cb; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;"></a></div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptal" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="mobile-javascript-frameworks" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Mobile JavaScript Frameworks</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As mobile web developers tend to face many of the same common challenges – such as cross-browser compatibility and inconsistent HTML and CSS in mobile browsers – frameworks have been developed (based on HTML5 and CSS3) that are specifically designed to address these issues and to work as flawlessly as possible on a wide array of smart phones and tablets. Most of these frameworks are lightweight, which helps facilitate fast mobile web browsing without compromising the look and feel of your site.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Broadening our view beyond the mobile landscape, if there is a single popular JavaScript framework worth mentioning, it is <a href="http://jquery.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">jQuery</a>. If you’re familiar with the desktop version, I recommend trying <a href="http://jquerymobile.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">jQuery Mobile</a> for your mobile web app. It has a widget library that converts semantic markup into a gesture-friendly format, making operations easy on touch-screens. The latest version consists of a really lightweight code base that packs a punch with a lot of graphical elements that really can improve your UI.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another alternative, <a href="http://www.sencha.com/products/touch" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Sencha Touch</a>, is rapidly gaining market share as well. It offers excellent performance overall and helps produce a mobile web user interface that largely looks and feels like a native one. Its full-featured widget library is based on Sencha’s <a href="http://www.sencha.com/products/extjs/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ExtJS</a> JavaScript library.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here are some key differences to consider when comparing jQuery Mobile and Sencha Touch:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Look and feel.</em></span> Generally speaking, the look and feel of a Sencha Touch app is crisper and superior to that of a jQuery mobile app, but it is important to remember that such reactions do tend to be highly subjective.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Extensibility.</em></span> jQuery Mobile offers lots of 3rd party extensions and is inherently designed to be highly extensible, whereas Sencha Touch is currently much more of a “closed” framework.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Device support.</em></span> jQuery Mobile currently targets a larger cross-section of devices than Sencha Touch.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">HTML “vs.” JavaScript.</em></span> jQuery is largely HTML-centric (i.e., extending and manipulating existing HTML in JavaScript), whereas Sencha Touch coding is entirely JavaScript-based. (This is an example, incidentally, of the skill set of your development team being important to consider when making your technology selections.)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">External dependencies.</em></span> jQuery mobile requires jQuery and jQuery UI for DOM manipulation, whereas Sencha Touch has no external dependencies.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Learning curve.</em></span> Most developers find the ramp-up time with jQuery to be less than that of Sencha Touch, perhaps fueled by the large percentage of web developers who are already familiar with the standard jQuery libraries.</li>
</ul>
<h3 id="responsive-frameworks" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Responsive Frameworks</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
An increasing number of responsive frameworks have begun cropping up in recent years, with two of the currently most popular being <a href="http://getbootstrap.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Bootstrap</a> and <a href="http://foundation.zurb.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Foundation</a>. In short, responsive frameworks simplify and streamline <a href="https://www.toptal.com/front-end/introduction-to-responsive-web-design-pseudo-elements-media-queries" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">web-based responsive UI design</a> and implementation, encapsulating the most common layouts and UI paradigms into a reusable, performance-optimized framework. Mostly based on CSS and JavaScript, many of these frameworks are open-source, free to download, and easily customizable. Unless you have a highly peculiar set of requirements, it is likely that use of one of these frameworks will reduce the level-of-effort to design and implement your mobile web application.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Examining the two leading options, Bootstrap and Foundation, a few of the key differences to consider include:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Targeted platforms.</em></span> While Bootstrap does support mobile, tablet, and desktop devices, it is primarily oriented toward desktop use. Foundation, on the other hand, is designed for essentially all screen sizes and types.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Browser compatibility.</em></span> Bootstrap is compatible with IE7 or higher, whereas Foundation is only compatible with IE9 or higher.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Diversity of layouts and components.</em></span> Bootstrap has a significantly larger collection of UI elements than is offered by Foundation.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Auto-resizing.</em></span> With Foundation, the grid shrinks and stretches according to the current browser height and width, whereas Bootstrap only supports a pre-defined set of grid sizes based on a standard set of screen sizes.</li>
</ul>
<h2 id="testing-and-debugging-your-mobile-web-app" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Testing and debugging your mobile web app</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Debugging mobile web apps can be tricky and somewhat frustrating, especially if you need to scrounge around for different devices to test on, or install SDKs for a (typically imperfect) emulation of the targeted client platforms.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this context, one clear advantage of mobile web app development (as compared with native app development) is that you can utilize standard browser-based developer tools to debug your application. Based on my personal preference for remote debugging, the one I recommend in this app development tutorial is Chrome with its DevTools. Other standard options include Firefox with Firebug or Opera’s Dragonfly tools.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="When learning how to develop web applications, look toward Chrome and its DevTools." src="https://assets.toptal.io/uploads/blog/image/313/toptal-blog-image-1394551497099.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Some of the reasons I prefer <a href="https://developers.google.com/chrome-developer-tools/docs/remote-debugging" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Chrome with its DevTools</a> include:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Mobile emulator in Chrome’s DevTools.</em></span> This is perhaps alone sufficient reason to select Chrome for debugging of mobile web apps. Key features include emulation of touch events, user agent spoofing, network bandwidth throttling, geolocation overrides, device orientation overrides, and CSS Media Type Emulation.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Interactive editor.</em></span> Ability to edit JavaScript or CSS on-the-fly.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Superior JavaScript debugger.</em></span> Allows for DOM breakpoints and provides the ability to profile your JavaScript code execution time.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Built-in JSON and XML viewers.</em></span> Avoids the need for any plugins to inspect server responses.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Support for the Android Debug Bridge (ADB) protocol directly over USB.</em></span> Facilitates easy instantiation of a remote debugging session. (<a href="https://developers.google.com/chrome-developer-tools/docs/remote-debugging" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Here</a> is a good tutorial by Google on how to start remotely debugging in Chrome.)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Dynamic inspection of resources.</em></span> Allows you to inspect your app’s local data sources, including IndexedDB or Web SQL databases, local and session storage, cookies, and Application Cache resources. You can also quickly inspect your application’s visual resources, including images, fonts, and style sheets.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To test the layout and cross browsing compatibility of your web app, you can also use some helpful online tools, such as <a href="http://www.browserstack.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">BrowserStack</a>. Just enter the URL for your application, select the browser, version, and operating system, and you’ll get the emulated view (and load speed) of your site in that environment. Another useful tool for the this purposes is <a href="https://crossbrowsertesting.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">CrossBrowserTesting</a>.</div>
<h2 id="wrap-up" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Wrap up</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With the continued rapid expansion of the number, variety and sophistication of mobile devices on the market and in use today, the need for effective, user-friendly, high performance mobile applications is likely to increase substantially. Being able to develop these applications intelligently and efficiently will therefore continue to be of paramount importance.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Many factors must be considered when choosing between the web, native, and hybrid options for mobile applications. Each has its own advantages, but mobile web apps will often represent your most efficient development (and therefore time-to-market) option. Should you choose to go down that path, I hope this web app development tutorial helps get you more directly and successfully to your destination.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <span style="color: #929292; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; text-transform: uppercase;"> </span><span style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/resume/tomas-agrimbau" style="border: 0px; box-sizing: border-box; color: #103d77; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">TOMAS AGRIMBAU</a>, </span><span style="font-size: 19.2px; line-height: 28.8px;">a <a href="https://www.toptal.com/android/developing-mobile-web-apps-when-why-and-how" target="_blank">Toptal</a><a href="https://www.toptal.com/mobile" target="_blank"> freelance developer</a>.</span></div>
Anonymousnoreply@blogger.com42tag:blogger.com,1999:blog-8706416286757464743.post-22330448182435270572016-08-10T22:04:00.000+07:002016-08-10T22:04:23.659+07:00Social Network APIs: The Internet’s Portal to the Real World<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Social network APIs have come a long way since Google released the first version of its YouTube API in May 2008 and Facebook released the first version of the Graph API in April 2010. Today, these APIs give you the opportunity to query social network platforms for posts, users, channels, and demographic data. They even let you create your own service or find out more about your user base.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this article, we will examine the ways we can utilize some of the popular social network APIs:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Facebook (Graph and Marketing API)</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Instagram</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Twitter</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">YouTube</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Pinterest</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We will also discuss their limitations, and explore some of the useful tools that are available for use with these APIs. Later in the article, we will also take a look at how to integrate these APIs in any <a href="https://www.toptal.com/ruby-on-rails" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Rails</a> application.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Social Network APIs: The Internet’s Portal to the Real World" src="https://assets.toptal.io/uploads/blog/image/92810/toptal-blog-image-1470067512236-de1fc92b07031fed0add24ecfca4b313.jpg" style="background: transparent; border-radius: 0px; border: 0px; box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Use social network APIs to get to know your users better than they know themselves.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I will focus on one social network API at a time and explain its capabilities, limitations and available tools. There will be a matrix with the different APIs and their properties for better comparison later in this article.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to use the APIs you will first need to setup an app that creates queries on behalf of your application with OAuth based requests. Users will authenticate against your app and you can then access their data with the resulting user access-token.</div>
<h2 id="facebook" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
FACEBOOK</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The now outdated FQL (Facebook Query Language) used to be a SQL-like query language that could be used to access all data from Facebook.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Facebook released the first version of its Graph API in April 2010. The most recent version at the time of writing this article is 2.6 which was introduced on April 12, 2016. It is a low level HTTP-based API that can be used to query data, create posts, and even create automated ad campaigns.</div>
<h3 id="tools" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tools</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://developers.facebook.com/tools/explorer/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Graph API Explorer</a> is the most commonly used tool when working with the Facebook API. It lets you execute Graph API queries in the browser and examine the results: You can use one of your app’s access tokens or create one on the fly with selected scopes.</div>
<h3 id="capabilities" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Capabilities</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Graph API is a REST-based API that lets you create, update, and delete objects per HTTP request on certain nodes.</div>
<h4 id="access-token" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Access Token</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To run queries against the Graph API, you need an access token that is obtained as soon as a user successfully authorizes in your app. The access token should be stored by your application.</div>
<h4 id="scopes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Scopes</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Scopes determine which actions can be performed on behalf of a user. The application asks for certain scopes when a user authorizes in an app. The <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">publish_actions</code> scope, for example, lets an app publish posts on behalf of a user. The email scope lets the app read the user’s email. A full overview over all scopes is <a href="https://developers.facebook.com/docs/facebook-login/permissions" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">listed in the official documentation</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Certain scopes like the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">publish_actions</code> or <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ads_management</code> require a review by Facebook prior to the release of the app.</div>
<h3 id="examples" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Examples</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To demonstrate how the Graph API works, I will show you how to read, create, update, and delete posts with the API.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get your own posts, you can execute the GET query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">/me/posts</code>. The result will be a JSON string with a list of posts, including their message, created_time, and id.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get more data about your posts, you can extend the query with fields as query parameters. For example, the query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">me/posts?fields=reactions, picture</code> will give you the post’s picture and reactions.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To create a post, you can simply send a POST action against the edge feed, e.g. <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">me/feed</code>, with parameters such as <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">message: hello world</code>. The Graph API will return a JSON object with the ID of your created post. You can then view the post at the address <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">http://facebook.com/[post_id]</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To update a post, you can send a POST request to the post’s node with the fields to be updated as parameters; e.g., <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">/[post_id]</code> and params like <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Message: lorem ipsum</code>. A success indicator with a value of true or false will be returned.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To delete a post, you can simply make a DELETE request to the node with the post ID (e.g., <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">/[post_id]</code>). The return value will be a JSON object with a success value of true or false.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A full overview over all nodes and actions is available in the <a href="https://developers.facebook.com/docs/graph-api/reference" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Graph API Reference</a>.</div>
<h4 id="marketing-api" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Marketing API</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://developers.facebook.com/docs/marketing-api" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">marketing API</a> deserves special mention because it is a powerful tool to manage Facebook ads and get ad insights through your application.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It works the same way as other Graph API methods. However, you need the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ads_management</code> scope in order to get access to the user’s ads. Facebook also needs to review your app before you can publish it.</div>
<h3 id="testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Testing</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once you create your app, it is in development mode and automatically visible in your app dashboard (i.e., <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">https://developers.facebook.com/apps/</code>).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In development mode, only admins, devs, and testers have access to your app. You can add testers and admins in the roles section of your app dashboard.</div>
<h3 id="review-process" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Review Process</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When adding certain permissions,Facebook needs to review your app before you can publish it. The review process is defined by <a href="https://developers.facebook.com/docs/apps/review" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">this set of guidelines</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to submit certain items for review, you can simply add them in the App Review section of your app dashboard. Facebook will then guide you through the review process and you will be alerted once your app is approved.</div>
<h3 id="limitations-and-workarounds" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Limitations and Workarounds</h3>
<h4 id="rate-limits" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Rate Limits</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
An app can make 200 calls per hour per user in aggregate. If you reach that limit, your API calls will result in error.</div>
<h4 id="searching-for-posts-on-facebook" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Searching for Posts on Facebook</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Facebook restricts searching for posts and tags on Facebook through the Graph API and FQL. However, you can use the <a href="https://developers.google.com/web-search/docs/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Google Search API</a> to search for public Facebook posts and then use the post-id in the URL to retrieve more information about specific posts through the Graph API.</div>
<h4 id="getting-custom-audience-data" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Getting Custom Audience Data</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://www.facebook.com/ads/audience-insights/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Audience Insights</a> on Facebook is a powerful research tool to learn more about a particular audience based on interests, demographics, or a other attrributes (e.g., a collection of email addresses).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, I have not found a way to automatically create audience insights through the ad API. Let us know in the comments if you have any creative ideas or suggestions for this.</div>
<h2 id="instagram" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
INSTAGRAM</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92802/toptal-blog-image-1469716070108-faf64525ae5120791b4cfbe6ec2bc9a6.jpg" style="background: transparent; border-radius: 0px; border: 0px; box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://www.instagram.com/developer/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Instagram API</a> was first released in April 2014 and allows you to build apps that analyze user posts and help users to manage their own posts.</div>
<h3 id="tools-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tools</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since the API console by Instagram is <a href="https://www.instagram.com/developer/deprecated/api-console/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">deprecated</a> at the time of this article, I recommend using <a href="https://apigee.com/console/instagram?apig_cc=1" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Apigee</a> for testing purposes in your browser.</div>
<h3 id="capabilities-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Capabilities</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Instagram API is a REST-based API. All of its <a href="https://www.instagram.com/developer/endpoints/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">endpoints</a> are described in their official documentation.</div>
<h4 id="access-token-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Access Token</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To run queries against the Instagram API, you need an <a href="https://www.instagram.com/developer/authentication/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">access token</a> that is obtained as soon as a user authorizes in your app. In order for a user to receive an access token, he or she must be directed to your app’s authorization URL. The server will then redirect the user after authorizing your app and you will then be able to read the token.</div>
<h4 id="scopes-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Scopes</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Your app can ask for different <a href="https://www.instagram.com/developer/authorization/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">permissions</a>. For instance, “basic” limits you to reading a user’s profile info and media. “public_content” lets you read any public profile and media on behalf of a user.</div>
<h4 id="examples-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Examples</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To demonstrate how the Instagram API works, I will go through some examples based on the media endpoint <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">https://api.instagram.com/v1/media/popular</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This endpoint returns the currently popular media from Instagram if passed an access token as a parameter. The result will be a JSON array of posts containing, for each, its media ID, a link to its image, likes, comments, the user that posted it, and some other attributes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can use apigee to play around and find out more about the API endpoints and their parameters.</div>
<h3 id="testing-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Testing</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Every new app created on the Instagram platform starts in <a href="https://www.instagram.com/developer/sandbox/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">sandbox mode</a>. This is a fully functional environment that allows you to test publicly available API endpoints before you submit your app for review.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To test your app, simply create a staging version and run all queries through that version instead of the live version that got through the review.</div>
<h3 id="review-process-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Review Process</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Apps in sandbox mode can use any API endpoint, but are restricted to a limited number of users and media. It’s a great mechanism for developing and testing an app.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To go live and access all Instagram content, you will need to submit your application for <a href="https://www.instagram.com/developer/review/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">review</a>. Once reviewed, you will only be able to request the scopes for users for which your app was approved.</div>
<h3 id="limitations-and-workarounds-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Limitations and workarounds</h3>
<h4 id="demographic-analysis" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Demographic Analysis</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
At the time of writing this article, there is no way to get information about a public users’ age, gender, or interests, because Instagram does not provide you with that information.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to get demographics data about followers or a list of Instagram users, you would need to iterate over all of them and try to determine their age and gender or interests based on their followers or the information provided in their bio.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A good big data solution for this problem might be a valuable service to some companies.</div>
<h4 id="rate-limits-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Rate Limits</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All <a href="https://www.instagram.com/developer/limits/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">rate limits</a> on the Instagram platform are controlled by access token on a sliding 1-hour window. Live apps have higher rate limits than apps in Sandbox Mode. The global rate limit for a live app is currently 5,000 calls per hour.</div>
<h2 id="twitter" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
TWITTER</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Twitter API was <a href="https://blog.twitter.com/2006/introducing-the-twitter-api" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">first released</a> in September 2006. It is a <a href="https://dev.twitter.com/rest/public" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">public REST API</a> that provides read and write access to Twitter data. Authentication is performed using OAuth. Responses are in JSON format.</div>
<h3 id="tools-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tools</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Twitter has an <a href="https://dev.twitter.com/rest/tools/console" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">API console tool</a> powered by apigee that can be used to test requests in the browser.</div>
<h3 id="capabilities-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Capabilities</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The REST API lets you get a user’s tweets, followers, and followed people. You can also search for hashtags in other tweets.</div>
<h4 id="access-token-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Access Token</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Twitter lets you create apps that users can <a href="https://dev.twitter.com/oauth" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">authenticate</a> against in return of an access token. The authentication model is OAuth.</div>
<h4 id="scopes-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Scopes</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are only two permissions that have to be set on the app’s setting page: Read only and Read and Write. The latter lets you create tweets and perform other post actions on behalf of a user.</div>
<h4 id="examples-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Examples</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To demonstrate the usage of the <a href="https://api.twitter.com/1.1/statuses/user_timeline.json" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Twitter API</a> I will retrieve the authorized user’s tweets. The result is a JSON array with the tweet’s images, favorites, retweets, urls, creation date, and other attributes. Use Apigee to play around and find out more about the API endpoints and their parameters.</div>
<h3 id="testing-and-review-process" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Testing and Review Process</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is currently no review process or test mode available for the Twitter API.</div>
<h3 id="limitations-and-workarounds-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Limitations and Workarounds</h3>
<h4 id="demographic-analysis-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Demographic Analysis</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is currently no easy way to get demographic data from someone’s Twitter followers. The brute force approach would be to browse through each follower and try to get the data through their bio and linked social network accounts.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can then make further assumptions based on the collected follower data through data analysis. Another way to get more insights is through Twitter’s paid enterprise API platform <a href="https://gnip.com/about/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">GNIP</a>. Among other things, it lets you create audiences and get information about those through the API. The API is currently in BETA.</div>
<h4 id="rate-limits-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Rate Limits</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Twitter has <a href="https://dev.twitter.com/rest/public/rate-limiting" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">rate limits</a> on a per-user basis and on a 15 minute basis. If your application has multiple tokens, you can simply alternate tokens for public operations in order to avoid reaching the limit.</div>
<h2 id="youtube" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
YOUTUBE</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92803/toptal-blog-image-1469716087202-70bfcc6f6800c858c94f94261a94b994.jpg" style="background: transparent; border-radius: 0px; border: 0px; box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The YouTube Data API was first introduced in January 2013. It lets you add YouTube features to your application, search for content, and analyze a YouTube channel’s demographics. It is an OAuth, token-based REST API that returns JSON responses.</div>
<h3 id="tools-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tools</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://developers.google.com/apis-explorer/#p/youtube/v3/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">API Explorer</a> lets you test unauthorized and authorized requests. You can run requests from your browser against the provided endpoints.</div>
<h3 id="capabilities-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Capabilities</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Among other things, you can work with activities, chats, live broadcasts, playlists, channels, videos, and subscriptions. Most of the endpoints require you to authorize with a YouTube account.</div>
<h4 id="access-token-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Access Token</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The YouTube Data API supports the OAuth 2.0 protocol for <a href="https://developers.google.com/youtube/v3/guides/authentication" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">authorizing access</a> to private user data. Once a user has been authorized in your application, they will be redirected to your application where the access token should be saved.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to use OAuth 2.0 authorization, you first need to obtain authorization credentials in the Google developer console.</div>
<h4 id="scopes-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Scopes</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The YouTube Data API currently supports the following scopes:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://www.googleapis.com/auth/youtube.force-ssl" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Force SSL</a> - Manage your youtube account but only over an SSL connection.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://www.googleapis.com/auth/youtube" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Default</a> - Manage your YouTube account. This scope is functionally identical to the youtube.force-ssl scope but does not require an SSL connection.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://www.googleapis.com/auth/youtube.readonly" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Read Only</a> - View your YouTube account.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://www.googleapis.com/auth/youtube.upload" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Upload</a> - Upload YouTube videos and manage your YouTube videos.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://www.googleapis.com/auth/youtubepartner-channel-audit" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Partner Channel Audit</a> - Retrieve information that Multichannel Networks use as criteria to accept or reject a channel in their network.</li>
</ul>
<h4 id="examples-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Examples</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As an example of usage of the Youtube Data API, the following request queries for videos with “coding” in their title and description:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">https://www.googleapis.com/youtube/v3/search?part=snippet&q=coding&key={YOUR_API_KEY}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The result is a JSON object containing the title, description, videoId, and channelId. You can use the latter to find out more about the channel.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">part</code> parameter is required for any API request that returns a certain resource. The parameter identifies resource properties that should be included in an API response. For example, a video resource has the following parts: snippet, contentDetails, fileDetails, player, processingDetails, recordingDetails, statistics, status, suggestions, topicDetails.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All other parameters, except the API key, differ from call to call. Read more about it in the <a href="https://developers.google.com/youtube/v3/docs/#calling-the-api" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">API reference guide</a>.</div>
<h2 id="pinterest" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
PINTEREST</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="https://developers.pinterest.com/docs/getting-started/introduction/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Pinterest API</a> was initially released in April 2015. It is a RESTful API that provides access to a user’s Pinterest data, such as their boards, pins, followers and more. The Pinterest API uses OAuth and allows both read and write permissions when interacting with a user’s content.</div>
<h3 id="tools-4" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tools</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Like others, Pinterest provides an API Explorer to test their endpoints and run queries against them. You can have a look at all their tools <a href="https://developers.pinterest.com/tools" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">here</a>.</div>
<h3 id="capabilities-4" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Capabilities</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Pinterest <a href="https://developers.pinterest.com/docs/api/overview/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">REST API</a> allows you to create pins, boards and query Pinterest data with OAuth.</div>
<h4 id="access-token-4" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Access Token</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Pinterest uses OAuth 2.0 to authenticate requests between your app and your users. All requests must be made over HTTPS.</div>
<h4 id="scopes-4" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Scopes</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Scopes determine what an app can do on behalf of a user. Pinterest uses the following scopes:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">none</code> (must know the identifier): Use GET method on a user’s profile, board and Pin details, and the Pins on a board.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">read_public</code>: Use GET method on a user’s Pins, boards and likes.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">write_public</code>: Use PATCH, POST, and DELETE methods on a user’s Pins and boards.</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">read_relationships</code>: Use GET method on a user’s follows and followers (on boards, users and interests).</li>
<li style="border: 0px; box-sizing: border-box; color: #444444; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">write_relationships</code>: Use PATCH, POST, and DELETE methods on a user’s follows and followers (on boards, users and interests).</li>
</ul>
<h4 id="examples-4" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Examples</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To demonstrate the use of the Pinterest API, I will demonstrate how to read the user’s latest pins:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">https://api.pinterest.com/v1/me/pins/?access_token={your_token}&fields=id,link,note,url,counts,board,created_at</code>will return a user’s pins with their id, link, note, url, likes, and repins.</div>
<h3 id="testing-and-review-process-1" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Testing and Review Process</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Apps are initially in development mode and must be submitted for review before they are released in production mode.</div>
<h3 id="limitations-and-workarounds-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Limitations and workarounds</h3>
<h4 id="demographic-analysis-2" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Demographic Analysis</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is no common way to get demographics data from a board. However,you can try to get a board’s followers and information about them from their bio and links to other social network accounts. A big data solution over the user’s common connections would also be a possibility.</div>
<h4 id="search-for-pins" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Search for Pins</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is currently no way to search for pins with certain tags or keywords through the API. You can bypass that limitation by using the Google Custom Search API to search for results on Pinterest pins only and gather the pin ID through the URL. The ID can then be used to get information about the pin through the API.</div>
<h4 id="rate-limits-3" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Rate Limits</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Each app (with a unique app ID) is allowed 1,000 calls per endpoint per hour for each unique user token.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Every API response returns a header that gives you an update about rate limits. X-Ratelimit-Limit is the rate limit for that specific request, and X-Ratelimit-Remaining is the number of requests you have left in the 60-minute window.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you exceed your rate limit for a given endpoint, you’ll get a 429 “Too many requests” error code.</div>
<h2 id="comparison-of-social-network-apis" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMPARISON OF SOCIAL NETWORK APIS</h2>
<table class="comparison-table" style="background-color: white; border-collapse: collapse; border-spacing: 0px; border: 2px solid black; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><thead style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><th style="background-color: #426ba7; border: 1px solid black; box-sizing: border-box; color: white; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;"></th><th style="background-color: #426ba7; border: 1px solid black; box-sizing: border-box; color: white; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Version</th><th style="background-color: #426ba7; border: 1px solid black; box-sizing: border-box; color: white; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth</th><th style="background-color: #426ba7; border: 1px solid black; box-sizing: border-box; color: white; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Format</th><th style="background-color: #426ba7; border: 1px solid black; box-sizing: border-box; color: white; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Demographics</th></tr>
</thead><tbody style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Facebook</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">v2.6<br style="box-sizing: border-box;" /><i style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Initial Release: April 2010</i></td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth 2</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">REST requests with JSON responses</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Supported</td></tr>
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Instagram</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">v1<br style="box-sizing: border-box;" /><i style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Initial Release: April 2014</i></td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth 2</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">REST requests with JSON responses</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Not supported</td></tr>
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Twitter</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">v1.1<br style="box-sizing: border-box;" /><i style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Initial Release: September 2006</i></td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth 1</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">REST requests with JSON responses</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Only supported with GNIP</td></tr>
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">YouTube</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">v3<br style="box-sizing: border-box;" /><i style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Initial Release: January 2013</i></td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth 2</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">REST requests with JSON responses</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Supported</td></tr>
<tr style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Pinterest</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">v1<br style="box-sizing: border-box;" /><i style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Initial Release: April 2015</i></td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">OAuth 2</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">REST requests with JSON responses</td><td style="border: 1px solid black; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 10px; vertical-align: baseline;">Not Supported</td></tr>
</tbody></table>
<h2 id="demo-application-with-devise" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
DEMO APPLICATION WITH DEVISE</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Integrating these APIs in your new or existing applications, thanks to a plethora of social network API packages and libraries, is easier than ever. Most modern platforms and frameworks have time-tested third-party libraries that even unify the authentication aspect all these APIs into a single library with neat plugin architecture.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92804/toptal-blog-image-1469716116722-4c52ac98563f4ebb484d7b830624070d.jpg" style="background: transparent; border-radius: 0px; border: 0px; box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For this article, we will take a look at how <a href="https://github.com/plataformatec/devise" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Devise</a>, a Ruby gem, does this ever so elegantly for Rails applications. Devise is a flexible authentication library based on <a href="https://rubygems.org/gems/warden/versions/1.2.6" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Warden</a> that implements authentication, registration, login, and data storage for multiple login providers. If you are more of a front-end guy and want to check something similar out for AngularJS, take a look at <a href="https://www.toptal.com/angular-js/facebook-login-angularjs-app-satellizer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">this article</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Devise, like most libraries of this class, doesn’t come built-in with support for any of the above mentioned social network APIs. Support for each of these social network API is provided through additional gems. The following gems are available for Rails authentication that cover the 5 providers discussed in this article:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">gem 'omniauth-facebook'
gem 'omniauth-pinterest'
gem 'omniauth-twitter'
gem 'omniauth-google-oauth2'
gem 'omniauth-instagram'</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="background-color: #fafafa; color: #505050; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">One of the best things about Rails is that there are many plugins written by the open source community. These are distributed as gems. Listed under a central configuration file, these gems are managed by Bundler.</span></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since these only provide authentication, registration, login, and storage for each of those providers, we will also need to get the following gems for the actual API clients:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">gem 'twitter' # https://github.com/sferik/twitter
gem 'instagram' # https://github.com/facebookarchive/instagram-ruby-gem
gem 'koala' # (Facebook API) https://github.com/arsduo/koala
gem 'google-api-client' # (YouTube API), https://github.com/google/google-api-ruby-client
gem 'pinterest-api' # https://github.com/realadeel/pinterest-api
</code></pre>
<h3 id="omniauth-and-authentication" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Omniauth and Authentication</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order for a user to authorize your app with your provider, you can simply provide a link with the following path:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">omniauth_authorize_path(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'user'</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:facebook</span>)
omniauth_authorize_path(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'user'</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:instagram</span>)
...
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to react on the callback after authenticating a user you can define a OmniauthCallbacksController with the scopes as functions like:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">AuthenticationsController</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Devise::OmniauthCallbacksController</span></span></span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>facebook
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> request.env[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"omniauth.auth"</span>]
...
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
That is the place to add a new Authentication model with the token and data into your application:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">authentication = where(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">provider:</span> omniauth.provider, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">user_id:</span> user.id)
.first_or_create <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |auth|
auth.user = user
auth.uid = omniauth.uid
auth.secret = omniauth.credentials.secret
auth.token = omniauth.credentials.token
...
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<h3 id="making-api-calls" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Making API Calls</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here is an example of how to use <a href="https://github.com/arsduo/koala" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Koala</a> to query the Facebook API. The rest of the providers work more or less similarly and are documented in the gem’s README.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is how you get your user data using Koala:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">authentication = user.authentication_for_provider(:facebook)
token = authentication.token
api = Koala::Facebook::API.new(token)
results = api.get_object("me")
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can then use the JSON result returned by the API. Source code of this demo application is <a href="https://github.com/Behsaad/social_media_apis_demo_public" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">available on GitHub</a>.</div>
<h2 id="wrap-up" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
WRAP UP</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Social network APIs provide you with a powerful tool to query the large data set of social networks and collect big data for your application. You can build a service on top of these APIs or use them to enhance your own application and user insights.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Rails and the available gems make it easy to integrate these APIs into your rails app and query the interfaces with an abstraction layer between your app and the API.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/behsaad-ramez" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Behsaad Ramez</a>, a <a href="https://www.blogger.com/" style="color: #4d469c; text-decoration: none;"><span id="goog_581781099"></span>Toptal<span id="goog_581781100"></span></a><a href="https://www.toptal.com/ruby" style="color: #4d469c; text-decoration: none;" target="_blank"> Ruby developer</a>.</div>
Anonymousnoreply@blogger.com4tag:blogger.com,1999:blog-8706416286757464743.post-66312964549000451792016-08-04T19:24:00.000+07:002016-08-04T19:24:21.125+07:00Write Tests That Matter: Tackle The Most Complex Code First<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are a lot of discussions, articles, and blogs around the topic of code quality. People say - use Test Driven techniques! Tests are a “must have” to start any refactoring! That’s all cool, but it’s 2016 and there is a massive volume of products and code bases still in production that were created ten, fifteen, or even twenty years ago. It’s no secret that a lot of them have legacy code with low test coverage.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
While I’d like to be always at the leading, or even bleeding, edge of the technology world - engaged with new cool projects and technologies – unfortunately it’s not always possible and often I have to deal with old systems. I like to say that when you develop from scratch, you act as a creator, mastering new matter. But when you’re working on legacy code, you’re more like a surgeon – you know how the system works in general, but you never know for sure whether the patient will survive your “operation”. And since it’s legacy code, there are not many up to date tests for you to rely on. This means that very frequently one of the very first steps is to cover it with tests. More precisely, not merely to provide coverage, but to develop a test coverage strategy.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Coupling and Cyclomatic Complexity: Metrics for Smarter Test Coverage" src="https://assets.toptal.io/uploads/blog/image/92790/toptal-blog-image-1469463837817-2e9e74c78ea4a99309408b33020036fe.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Forget 100% coverage. Test smarter by identifying classes that are more likely to break.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Basically, what I needed to determine was what parts (classes / packages) of the system we needed to cover with tests in the first place, where we needed unit tests, where integration tests would be more helpful etc. There are admittedly many ways to approach this type of analysis and the one that I’ve used may not be the best, but it’s kind of an automatic approach. Once my approach is implemented, it takes minimal time to actually do the analysis itself and, what is more important, it brings some fun into legacy code analysis.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The main idea here is to analyse two metrics – coupling (i.e., afferent coupling, or CA) and complexity (i.e. cyclomatic complexity).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first one measures how many classes use our class, so it basically tells us how close a particular class is to the heart of the system; the more classes there are that use our class, the more important it is to cover it with tests.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On the other hand, if a class is very simple (e.g. contains only constants), then even if it’s used by many other parts of the system, it’s not nearly as important to create a test for. Here is where the second metric can help. If a class contains a lot of logic, the Cyclomatic complexity will be high.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The same logic can also be applied in reverse; i.e., even if a class is not used by many classes and represents just one particular use case, it still makes sense to cover it with tests if its internal logic is complex.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is one caveat though: let’s say we have two classes – one with the CA 100 and complexity 2 and the other one with the CA 60 and complexity 20. Even though the sum of the metrics is higher for the first one we should definitely cover the second one first. This is because the first class is being used by a lot of other classes, but is not very complex. On the other hand, the second class is also being used by a lot of other classes but is relatively more complex than the first class.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To summarize: we need to identify classes with high CA and Cyclomatic complexity. In mathematical terms, a fitness function is needed that can be used as a rating - f(CA,Complexity) - whose values increase along with CA and Complexity.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">Generally speaking, the classes with the smallest differences between the two metrics should be given the highest priority for test coverage.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Finding tools to calculate CA and Complexity for the whole code base, and provide a simple way to extract this information in CSV format, proved to be a challenge. During my search, I came across two tools that are free so it would be unfair not to mention them:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Coupling metrics: <a href="http://www.spinellis.gr/sw/ckjm/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">www.spinellis.gr/sw/ckjm/</a></li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Complexity: <a href="http://cyvis.sourceforge.net/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">cyvis.sourceforge.net/</a></li>
</ul>
<h2 id="a-bit-of-math" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A Bit Of Math</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The main problem here is that we have two criteria – CA and Cyclomatic complexity – so we need to combine them and convert into one scalar value. If we had a slightly different task – e.g., to find a class with the worst combination of our criteria – we would have a classical multi-objective optimization problem:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92785/toptal-blog-image-1469451184452-f5dd2be391351d32ff966a5336f61219.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We would need to find a point on the so called Pareto front (red in the picture above). What is interesting about the Pareto set is that every point in the set is a solution to the optimization task. Whenever we move along the red line we need to make a compromise between our criteria – if one gets better the other one gets worse. This is called Scalarization and the final result depends on how we do it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are a lot of techniques that we can use here. Each has its own pros and cons. However, the most popular ones are <a href="https://en.wikipedia.org/wiki/Multi-objective_optimization" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">linear scalarizing</a> and the one based on an <a href="https://en.wikipedia.org/wiki/Multiple-criteria_decision_analysis" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">reference point</a>. Linear is the easiest one. Our fitness function will look like a linear combination of CA and Complexity:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
f(CA, Complexity) = A×CA + B×Complexity</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
where A and B are some coefficients.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The point which represents a solution to our optimization problem will lie on the line (blue in the picture below). More precisely, it will be at the intersection of the blue line and red Pareto front. Our original problem is not exactly an optimization problem. Rather, we need to create a ranking function. Let’s consider two values of our ranking function, basically two values in our Rank column:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
R1 = A∗CA + B∗Complexity and R2 = A∗CA + B∗Complexity</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Both of the formulas written above are equations of lines, moreover these lines are parallel. Taking more rank values into consideration we’ll get more lines and therefore more points where the Pareto line intersects with the (dotted) blue lines. These points will be classes corresponding to a particular rank value.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92786/toptal-blog-image-1469451831109-1610fdc86a6dda562250940aea17443e.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Unfortunately, there is an issue with this approach. For any line (Rank value), we’ll have points with very small CA and very big Complexity (and visa versa) lying on it. This immediately puts points with a big difference between metric values in the top of the list which is exactly what we wanted to avoid.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The other way to do the scalarizing is based on the reference point. Reference point is a point with the maximum values of both criteria:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
(max(CA), max(Complexity))</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The fitness function will be the distance between the Reference point and the data points:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
f(CA,Complexity) = √((CA−CA )<sup style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px;">2</sup> + (Complexity−Complexity)<sup style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px;">2</sup>)</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can think about this fitness function as a circle with the center at the reference point. The radius in this case is the value of the Rank. The solution to the optimization problem will be the point where the circle touches the Pareto front. The solution to the original problem will be sets of points corresponding to the different circle radii as shown in the following picture (parts of circles for different ranks are shown as dotted blue curves):</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92787/toptal-blog-image-1469451859809-d76868272924edfe9fed6b7bed1edad0.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This approach deals better with extreme values but there are still two issues: First – I’d like to have more points near the reference points to better overcome the problem that we’ve faced with linear combination. Second – CA and Cyclomatic complexity are inherently different and have different values set, so we need to normalize them (e.g. so that all the values of both metrics would be from 1 to 100).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here is a small trick that we can apply to solve the first issue – instead of looking at the CA and Cyclomatic Complexity, we can look at their inverted values. The reference point in this case will be (0,0). To solve the second issue, we can just normalize metrics using minimum value. Here is how it looks:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Inverted and normalized complexity – NormComplexity:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
(1 + min(Complexity)) / (1 + Complexity)∗100</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Inverted and normalized CA – NormCA:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
(1 + min(CA)) / (1+CA)∗100</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Note:</span> I added 1 to make sure that there is no division by 0.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The following picture shows a plot with the inverted values:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92788/toptal-blog-image-1469451934315-5a8e74c28e8c4e70100288e0487138f3.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="final-ranking" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Final Ranking</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We are now coming to the last step - calculating the rank. As mentioned, I’m using the reference point method, so the only thing that we need to do is to calculate the length of the vector, normalize it, and make it ascend with the importance of a unit test creation for a class. Here is the final formula:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: monospace; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
Rank(NormComplexity , NormCA) = 100 − √(NormComplexity<sup style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px;">2</sup> + NormCA<sup style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px;">2</sup>) / √2</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-email_form is-current" style="border: 0px; box-sizing: border-box; display: flex; flex-wrap: wrap; height: auto; margin: 0px; min-height: 0px; min-width: 0px; overflow: visible; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<br /></div>
</div>
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><br /><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="more-statistics" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
More Statistics</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is one more thought that I’d like to add, but let’s first have a look at some statistics. Here is a histogram of the Coupling metrics:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92789/toptal-blog-image-1469451950920-29066313892ee78898cb784dbbd31ff1.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What is interesting about this picture is the number of classes with low CA (0-2). Classes with CA 0 are either not used at all or are top level services. These represent <a href="https://www.toptal.com/api-developers" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">API</a> endpoints, so it’s fine that we have a lot of them. But classes with CA 1 are the ones that are directly used by the endpoints and we have more of these classes than endpoints. What does this mean from architecture / design perspective?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In general, it means that we have a kind of script oriented approach – we script every business case separately (we can’t really reuse the code as business cases are too diverse). If that is the case, then it’s definitely a <a href="https://en.wikipedia.org/wiki/Code_smell" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">code smell</a> and we need to do refactoring. Otherwise, it means the cohesion of our system is low, in which case we also need refactoring, but architectural refactoring this time.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Additional useful information we can get from the histogram above is that we can completely filter out classes with low coupling (CA in {0,1}) from the list of the classes eligible for coverage with unit tests. The same classes, though, are good candidates for the integration / functional tests.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can find all the scripts and resources that I have used in this GitHub repository: <a href="https://github.com/ashalitkin/code-base-stats" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ashalitkin/code-base-stats</a>.</div>
<h2 id="does-it-always-work" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Does It Always Work?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Not necessarily. First of all it’s all about static analysis, not runtime. If a class is linked from many other classes it can be a sign that it’s heavily used, but it’s not always true. For example, we don’t know whether the functionality is really heavily used by end users. Second, if the design and the quality of the system is good enough, then most likely different parts / layers of it are decoupled via interfaces so static analysis of the CA will not give us a true picture. I guess it’s one of the main reasons why CA is not that popular in tools like Sonar. Fortunately, it’s totally fine for us since, if you remember, we are interested in applying this specifically to old ugly code bases.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In general, I’d say that runtime analysis would give much better results, but unfortunately it’s much more costly, time consuming, and complex, so our approach is a potentially useful and lower cost alternative.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/andrey-shalitkin" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Andrey Shalitkin</a>, a <a href="https://www.toptal.com/software/coupling-and-cyclomatic-complexity" target="_blank">Toptal</a><a href="https://www.toptal.com/java" target="_blank"> Java developer</a>.</div>
Anonymousnoreply@blogger.com3tag:blogger.com,1999:blog-8706416286757464743.post-32516610485180283372016-07-29T10:18:00.000+07:002016-07-29T10:18:16.048+07:00How to Build a Multitenant Application: A Hibernate Tutorial<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When we talk about cloud applications where each client has their own separate data, we need to think about how to store and manipulate this data. Even with all the great NoSQL solutions out there, sometimes we still need to use the good old relational database. The first solution that might come to mind to separate data is to add an identifier in every table, so it can be handled individually. That works, but what if a client asks for their database? It would be very cumbersome to retrieve all those records hidden among the others.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Multitenancy Java EE Application with Hibernate" src="https://assets.toptal.io/uploads/blog/image/92781/toptal-blog-image-1468828880059-2c150232b4c11056be447cbf37d4215d.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Multitenancy in Java is easier than ever with Hibernate.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Hibernate team came up with a solution to this problem a while ago. They provide some extension points that enable one to control from where data should be retrieved. This solution has the option to control the data via an identifier column, multiple databases, and multiple schemas. This article will cover the multiple schemas solution.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, let’s get to work!</div>
<h2 id="getting-started" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Getting Started</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you are a more <a href="https://www.toptal.com/java" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">experienced Java developer</a> and know how to configure everything, or if you already have your own Java EE project, you can skip this section.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
First, we have to create a new Java project. I am using Eclipse and Gradle, but you can use your preferred IDE and building tools, such as IntelliJ and Maven.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you want to use the same tools as me, you can follow these steps to create your project:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Install <a href="https://marketplace.eclipse.org/content/gradle-sts-integration-eclipse" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Gradle plugin</a> on Eclipse</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click on File -> New -> Other…</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Find Gradle (STS) and click Next</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Inform a name and choose Java Quickstart for sample project</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Finish</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Great! This should be the initial file structure:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">javaee-mt
|- src/main/java
|- src/main/resources
|- src/test/java
|- src/test/resources
|- JRE System Library
|- Gradle Dependencies
|- build
|- src
|- build.gradle
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can delete all files that come inside the source folders, as they are just sample files.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To run the project, I use Wildfly, and I will show how to configure it (again you can use your favorite tool here):</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Download Wildfly: http://wildfly.org/downloads/ (I am using version 10)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Unzip the file</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Install the <a href="https://marketplace.eclipse.org/content/jboss-tools" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">JBoss Tools plugin</a> on Eclipse</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">On the Servers tab, right-click any blank area and choose New -> Server</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Choose Wildfly 10.x (9.x also works if 10 is not available, depending on your Eclipse version)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Next, choose Create New Runtime (next page) and click Next again</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Choose the folder where you unzipped Wildfly as Home Directory</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Finish</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, let’s configure Wildfly to know the database:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Go to the bin folder inside your Wildfly folder</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Execute add-user.bat or add-user.sh (depending on your OS)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Follow the steps to create your user as Manager</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">In Eclipse, go to the Servers tab again, right-click on the server you created and select Start</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">On your browser, access http://localhost:9990, which is the Management Interface</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Enter the credentials of the user you just created</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deploy the driver jar of your database:<ol style="border: 0px; box-sizing: border-box; font-size: 1em; list-style: none; margin: 1em 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Go to the Deployment tab and click Add</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Next, choose your driver jar file</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0px 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Next and Finish</li>
</ol>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Go to the Configuration tab</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Choose Subsystems -> Datasources -> Non-XA</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Add, select your database and click Next</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Give a name to your data source and click Next</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Select the Detect Driver tab and choose the driver you just deployed</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Enter your database information and click Next</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Test Connection if you want to make sure the information of the prior step is correct</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Finish</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Go back to Eclipse and stop the running server</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Right-click on it, select Add and Remove</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Add your project to the right</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Click Finish</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Alright, we have Eclipse and Wildfly configured together!</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is all the configurations required outside of the project. Let’s move on to the project configuration.</div>
<h2 id="bootstrapping-project" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Bootstrapping Project</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we have Eclipse and Wildfly configured and our project created, we need to configure our project.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first thing we are going to do is to edit build.gradle. This is how it should look:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-gradle" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">apply plugin: 'java'
apply plugin: 'war'
apply plugin: 'eclipse'
apply plugin: 'eclipse-wtp'
sourceCompatibility = '1.8'
compileJava.options.encoding = 'UTF-8'
compileJava.options.encoding = 'UTF-8'
compileTestJava.options.encoding = 'UTF-8'
repositories {
jcenter()
}
eclipse {
wtp {
}
}
dependencies {
providedCompile 'org.hibernate:hibernate-entitymanager:5.0.7.Final'
providedCompile 'org.jboss.resteasy:resteasy-jaxrs:3.0.14.Final'
providedCompile 'javax:javaee-api:7.0'
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The dependencies are all declared as “providedCompile”, because this command doesn’t add the dependency in the final war file. Wildfly already has these dependencies, and it would cause conflicts with the app’s ones otherwise.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
At this point, you can right-click your project, select Gradle (STS) -> Refresh All to import the dependencies we just declared.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Time to create and configure the “persistence.xml” file, the file that contains the information that Hibernate needs:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">In the src/main/resource source folder, create a folder called META-INF</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Inside this folder, create a file named persistence.xml</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The content of the file must be the something like the following, changing jta-data-source to match the datasource you created in Wildfly and the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">package com.toptal.andrehil.mt.hibernate</code> to the one you are going to create in the next section (unless you choose the same package name):</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-xml hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-pi" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><?xml version="1.0" encoding="UTF-8" ?></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">persistence</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">xmlns:xsi</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"http://www.w3.org/2001/XMLSchema-instance"</span>
<span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">xsi:schemaLocation</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"</span>
<span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">version</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"2.0"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">xmlns</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"http://java.sun.com/xml/ns/persistence"</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">persistence-unit</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"pu"</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">jta-data-source</span>></span>java:/JavaEEMTDS<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">jta-data-source</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">properties</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">property</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"hibernate.multiTenancy"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">value</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"SCHEMA"</span>/></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">property</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"hibernate.tenant_identifier_resolver"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">value</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"com.toptal.andrehil.mt.hibernate.SchemaResolver"</span>/></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">property</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"hibernate.multi_tenant_connection_provider"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">value</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"com.toptal.andrehil.mt.hibernate.MultiTenantProvider"</span>/></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">properties</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">persistence-unit</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">persistence</span>></span>
</code></pre>
<h2 id="hibernate-classes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Hibernate Classes</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The configurations added to persistence.xml point to two custom classes MultiTenantProvider and SchemaResolver. The first class is responsible for providing connections configured with the right schema. The second class is responsible for resolving the name of the schema to be used.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here is the implementation of the two classes:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiTenantProvider</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">implements</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MultiTenantConnectionProvider</span>, <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ServiceRegistryAwareService</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">private</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">long</span> serialVersionUID = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>L;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">private</span> DataSource dataSource;
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">supportsAggressiveRelease</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">injectServices</span>(ServiceRegistryImplementor serviceRegistry) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">try</span> {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Context init = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> InitialContext();
dataSource = (DataSource) init.lookup(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"java:/JavaEEMTDS"</span>); <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Change to your datasource name</span>
} <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">catch</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> NamingException e) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throw</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> RuntimeException(e);
}
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@SuppressWarnings</span>(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"rawtypes"</span>)
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">isUnwrappableAs</span>(Class clazz) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <T> T <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">unwrap</span>(Class<T> clazz) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> Connection <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getAnyConnection</span>() <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> SQLException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Connection connection = dataSource.getConnection();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> connection;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> Connection <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getConnection</span>(String tenantIdentifier) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> SQLException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Connection connection = getAnyConnection();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">try</span> {
connection.createStatement().execute(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"SET SCHEMA '"</span> + tenantIdentifier + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"'"</span>);
} <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">catch</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> SQLException e) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throw</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> HibernateException(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Error trying to alter schema ["</span> + tenantIdentifier + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"]"</span>, e);
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> connection;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">releaseAnyConnection</span>(Connection connection) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> SQLException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">try</span> {
connection.createStatement().execute(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"SET SCHEMA 'public'"</span>);
} <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">catch</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> SQLException e) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throw</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> HibernateException(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Error trying to alter schema [public]"</span>, e);
}
connection.close();
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">releaseConnection</span>(String tenantIdentifier, Connection connection) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> SQLException {
releaseAnyConnection(connection);
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The syntax being used in the statements above work with PostgreSQL and some other databases, this must be changed in case your database has a different syntax to change the current schema.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">SchemaResolver</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">implements</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">CurrentTenantIdentifierResolver</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">private</span> String tenantIdentifier = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"public"</span>;
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> String <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">resolveCurrentTenantIdentifier</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> tenantIdentifier;
}
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">validateExistingCurrentSessions</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">setTenantIdentifier</span>(String tenantIdentifier) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>.tenantIdentifier = tenantIdentifier;
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
At this point, it is already possible to test the application. For now, our resolver is pointing directly to a hard-coded public schema, but it is already being called. To do this, stop your server if it is running and start it again. You can try to run it in debug mode and place breakpoint at any point of the classes above to check if it is working.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-email_form is-current" style="border: 0px; box-sizing: border-box; display: flex; flex-wrap: wrap; height: auto; margin: 0px; min-height: 0px; min-width: 0px; overflow: visible; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<br /></div>
</div>
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><br /><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="practical-use-of-the-resolver" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Practical Use Of The Resolver</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, how could the resolver actually contain the right name of the schema?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One way to achieve this is to keep an identifier in the header of all requests and then create a filter to inject the name of the schema.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s implement a filter class to exemplify the usage. The resolver can be accessed through Hibernate’s SessionFactory, so we will take advantage of that to get it and inject the right schema name.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Provider</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">AuthRequestFilter</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">implements</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ContainerRequestFilter</span> {</span>
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@PersistenceUnit</span>(unitName = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"pu"</span>)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">private</span> EntityManagerFactory entityManagerFactory;
<span class="hljs-annotation" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@Override</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">filter</span>(ContainerRequestContext containerRequestContext) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> IOException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> SessionFactoryImplementor sessionFactory = ((EntityManagerFactoryImpl) entityManagerFactory).getSessionFactory();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> SchemaResolver schemaResolver = (SchemaResolver) sessionFactory.getCurrentTenantIdentifierResolver();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> String username = containerRequestContext.getHeaderString(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"username"</span>);
schemaResolver.setTenantIdentifier(username);
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, when any class gets an EntityManager to access the database, it will be already configured with the right schema.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For the sake of simplicity, the implementation shown here is getting the identifier directly from a string in the header, but it is a good idea to use an authentication token and store the identifier in the token. If you are interested in knowing more about this subject, I suggest taking a look at JSON Web Tokens (JWT). JWT is a nice and simple library for token manipulation.</div>
<h2 id="how-to-use-all-of-this" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How to Use All of This</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With everything configured, there is nothing else needed to do in your entities and/or classes that interact with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">EntityManager</code>. Anything you run from an EntityManager will be directed to the schema resolved by the created filter.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, all you need to do is to intercept requests on the client side and inject the identifier/token in the header to be sent to the server side.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">In a real application, you will have a better means of authentication. The general idea of multitenancy, however, will remain the same.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The link at the end of the article points to the project used to write this article. It uses Flyway to create 2 schemas and contains an entity class called Car and a rest service class called <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CarService</code> that can be used to test the project. You can follow all the steps below, but instead of creating your own project, you can clone it and use this one. Then, when running you can use a simple HTTP client (like Postman extension for Chrome) and make a GET request to http://localhost:8080/javaee-mt/rest/cars with the headers key:value:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">username:joe; or</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">username:fred.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By doing this, the requests will return different values, which are in different schemas, one called joe and the other one called “fred”.</div>
<h2 id="final-words" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Final Words</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is not the only solution to create multitenancy applications in the Java world, but it is a simple way to achieve this.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One thing to keep in mind is that Hibernate doesn’t generate DDL when using multitenancy configuration. My suggestion is to take a look at Flyway or Liquibase, which are great libraries to control database creation. This is a nice thing to do even if you are not going to use multitenancy, as the Hibernate team advises to not use their auto database generation in production.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The source code used to create this article and environment configuration can be found at <a href="https://github.com/andrehil/JavaEEMT" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">github.com/andrehil/JavaEEMT</a></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/andre-william-prade-hildinger" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">André William Prade Hildinger</a>, a <a href="https://www.toptal.com/hibernate/build-multitenant-java-hibernate" target="_blank">Toptal</a><a href="https://www.toptal.com/javascript" target="_blank"> Javascript developer</a>.</div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-23740924168429075462016-07-22T06:10:00.002+07:002016-07-22T06:10:36.364+07:00Hunting Down Memory Issues In Ruby: A Definitive Guide<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
I’m sure there are some lucky <a href="https://www.toptal.com/ruby" target="_blank">Ruby developers</a> out there who will never run into issues with memory, but for the rest of us, it’s incredibly challenging to hunt down where memory usage is getting out of hand and fix it. Fortunately, if you’re using a modern Ruby (2.1+), there are some great tools and techniques available for dealing with common issues. It could also be said that memory optimization can be fun and rewarding although I may be alone in that sentiment.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Hunting Down Memory Issues In Ruby" src="https://assets.toptal.io/uploads/blog/image/92743/toptal-blog-image-1467725322746-6816d91a0142ba0b220dcd01e3dd1202.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<br />
<div style="text-align: justify;">
If you thought bugs were pesky, wait until you hunt for memory issues.</div>
<div style="text-align: justify;">
<iframe allowtransparency="true" class="twitter-share-button twitter-share-button-rendered twitter-tweet-button" data-url="https://www.toptal.com/ruby/hunting-ruby-memory-issues" frameborder="0" id="twitter-widget-0" scrolling="no" src="https://platform.twitter.com/widgets/tweet_button.a9003d9964444592507bbb36b98c709b.en.html#dnt=false&id=twitter-widget-0&lang=en&original_referer=https%3A%2F%2Fwww.toptal.com%2Fruby%2Fhunting-ruby-memory-issues&size=m&text=If%20you%20thought%20bugs%20were%20pesky%2C%20wait%20until%20you%20hunt%20for%20memory%20issues.&time=1469142506656&type=share&url=https%3A%2F%2Fwww.toptal.com%2Fruby%2Fhunting-ruby-memory-issues" style="border-width: 0px; box-sizing: border-box; height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: static; vertical-align: baseline; visibility: visible; width: 93px !important;" title="Twitter Tweet Button"></iframe></div>
<div style="text-align: justify;">
It’s Not a Memory Leak!</div>
<code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># common.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"active_record"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"active_support/all"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"get_process_mem"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sqlite3"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ActiveRecord::Base</span>.establish_connection(</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">adapter:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sqlite3"</span>,</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">database:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"people.sqlite3"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< <span class="hljs-parent" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ActiveRecord::Base</span></span>;</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>print_usage(description)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
mb = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">GetProcessMem</span>.new.mb</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
puts <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{ description }</span> - MEMORY USAGE(MB): <span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{ mb.round }</span>"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>print_usage_before_and_after</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Before"</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">yield</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"After"</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>random_name</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>...<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">20</span>).map { (<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">97</span> + rand(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">26</span>)).chr }.join</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># build_arrays.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARRAY_SIZE</span> = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1_000_000</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
times = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARGV</span>.first.to_i</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>..times).each <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |n|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
foo = []</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARRAY_SIZE</span>.times { foo << {<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">some:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"stuff"</span>} }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage(n)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby build_arrays.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">17</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">330</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">481</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">492</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">4</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">559</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">5</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">584</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">6</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">588</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">7</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">591</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">8</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">603</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">9</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">613</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">621</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby build_arrays.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">40</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">9</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">323</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
...</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">32</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">700</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">33</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">699</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">34</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">698</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">35</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">698</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">696</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">37</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">696</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">38</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">696</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">39</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">701</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">40</span> - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">697</span></div>
</code><div style="text-align: justify;">
Do not panic if you see a sudden rise in the memory usage of your app. Apps can run out of memory for all sorts of reasons - not just memory leaks.</div>
<div style="text-align: justify;">
Divide and Conquer</div>
<div style="text-align: justify;">
Isolating Memory Usage Hotspots</div>
<code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># people.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>run(number)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.delete_all</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
names = number.times.map { random_name }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
names.each <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |name|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.create(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name:</span> name)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
records = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.all.to_a</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.open(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"people.txt"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"w"</span>) { |out| out << records.to_json }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># before_and_after.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./people"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
run(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARGV</span>.shift.to_i)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby before_and_after.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10000</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">37</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">96</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># profile.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"memory_profiler"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./people"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
report = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">MemoryProfiler</span>.report <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
run(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1000</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
report.pretty_print(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">to_file:</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"profile.txt"</span>)</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
allocated memory by gem</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
-----------------------------------</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
17520444 activerecord-4.2.6</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
7305511 activesupport-4.2.6</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2551797 activemodel-4.2.6</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2171660 arel-6.0.3</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2002249 sqlite3-1.3.11</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
...</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
allocated memory by file</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
-----------------------------------</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2840000 /Users/bruz/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.6/lib/activ</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
e_support/hash_with_indifferent_access.rb</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2006169 /Users/bruz/.rvm/gems/ruby-2.2.4/gems/activerecord-4.2.6/lib/active</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
_record/type/time_value.rb</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
2001914 /Users/bruz/code/mem_test/people.rb</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
1655493 /Users/bruz/.rvm/gems/ruby-2.2.4/gems/activerecord-4.2.6/lib/active</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
_record/connection_adapters/sqlite3_adapter.rb</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
1628392 /Users/bruz/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.6/lib/activ</div>
</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
e_support/json/encoding.rb</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># File.open("people.txt", "w") { |out| out << records.to_json }</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby before_and_after.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10000</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">47</span> MB</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># records = Person.all.to_a</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
records = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.all</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># File.open("people.txt", "w") { |out| out << records.to_json }</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby before_and_after.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10000</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">40</span> MB</div>
</code><div style="text-align: justify;">
Deserialization</div>
<div style="text-align: justify;">
Just because you have limited memory doesn't mean you cannot parse large XML or JSON documents safely. Streaming deserializers allow you to incrementally extract whatever you need from these documents and still keep the memory footprint low.</div>
<code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># parse_with_from_xml.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># From http://www.cs.washington.edu/research/xmldatasets/data/mondial/mondial-3.0.xml</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
file = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.open(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.expand_path(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"../mondial-3.0.xml"</span>, __FILE_<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">_</span>))</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
hash = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Hash</span>.from_xml(file)[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"mondial"</span>][<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"continent"</span>]</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
puts hash.map { |c| c[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"name"</span>] }.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">", "</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby parse_with_from_xml.rb</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">37</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Europe, Asia, America, Australia/Oceania, Africa</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">164</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># parse_with_ox.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"ox"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Handler</span> <span class="hljs-inheritance" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">< </span>::<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ox::Sax</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>initialize(&block)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@yield_to</span> = block</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>start_element(name)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">case</span> name</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">when</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:continent</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@in_continent</span> = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>end_element(name)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">case</span> name</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">when</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:continent</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@yield_to</span>.call(<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@name</span>) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@name</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@in_continent</span> = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@name</span> = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">nil</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>attr(name, value)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">case</span> name</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">when</span> <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:name</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@name</span> = value <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> <span class="hljs-variable" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">@in_continent</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># From http://www.cs.washington.edu/research/xmldatasets/data/mondial/mondial-3.0.xml</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
file = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.open(<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.expand_path(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"../mondial-3.0.xml"</span>, __FILE_<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">_</span>))</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
continents = []</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
handler = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Handler</span>.new <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |continent|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
continents << continent</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ox</span>.sax_parse(handler, file)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
puts continents.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">", "</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby parse_with_ox.rb</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">37</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Europe, Asia, America, Australia/Oceania, Africa</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After - MEMORY USAGE(MB): <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">37</span></div>
</code><div style="text-align: justify;">
Serialization</div>
<code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># to_json.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.open(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"people.txt"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"w"</span>) { |out| out << <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.all.to_json }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby to_json.rb</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">505</span> MB</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># json_stream.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"json-write-stream"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
file = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">File</span>.open(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"people.txt"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"w"</span>)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">JsonWriteStream</span>.from_stream(file) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |writer|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
writer.write_object <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |obj_writer|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
obj_writer.write_array(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"people"</span>) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |arr_writer|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Person</span>.find_each <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |people|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
arr_writer.write_element people.as_json</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby json_stream.rb</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">56</span> MB</div>
</code><div style="text-align: justify;">
Being Lazy</div>
<code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># not_lazy.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
number = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARGV</span>.shift.to_i</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
names = number.times</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { random_name }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { |name| name.capitalize }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { |name| <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{ name }</span> Jr."</span> }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.select { |name| name[<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>] == <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"X"</span> }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.to_a</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby not_lazy.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>_000_000</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">546</span> MB</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #93a1a1;"># lazy.rb</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
require_relative <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"./common"</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
number = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ARGV</span>.shift.to_i</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
print_usage_before_and_after <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
names = number.times.lazy</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { random_name }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { |name| name.capitalize }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.map { |name| <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{ name }</span> Jr."</span> }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.select { |name| name[<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>] == <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"X"</span> }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
.to_a</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
$ ruby lazy.rb <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>_000_000</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
Before: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">36</span> MB</div>
</code><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
After: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">52</span> MB</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> </span>records</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Enumerator</span>.new <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span> |yielder|</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
has_more = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
page = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">while</span> has_more</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
response = fetch(page)</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
response.records.each { |record| yielder << record }</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
page += <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
has_more = response.has_more</div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><code class="language-ruby hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><div style="text-align: justify;">
<span style="color: #859900;">end</span></div>
</code><div style="text-align: justify;">
Conclusion</div>
<br />
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As with all forms of optimization, odds are that it will add code complexity, so it’s not worth doing unless there are measurable and significant gains.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Everything described here is done using the canonical MRI Ruby, version 2.2.4, although other 2.1+ versions should behave similarly.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
When a memory issue is discovered, it’s easy to jump to the conclusion that there’s a memory leak. For example, in a web application, you may see that after you spin up your server, repeated calls to the same endpoint keep driving memory usage up higher with each request. There are certainly cases where legitimate memory leaks happen, but I’d wager they are vastly outnumbered by memory issues with this same appearance that aren’t actually leaks.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As a (contrived) example, let’s look at a bit of Ruby code that repeatedly builds a big array of hashes and discards it. First, here’s some code that’ll be shared throughout the examples in this post:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
And the array builder:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
The <a href="https://github.com/schneems/get_process_mem" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">get_process_mem</a> gem is just a convenient way to get the memory being used by the current Ruby process. What we see is the same behavior that was described above, a continual increase in memory usage.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
However, if we run more iterations, we’ll eventually plateau.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Hitting this plateau is the hallmark of not being an actual memory leak, or that the memory leak is so small that it’s not visible compared to the rest of the memory usage. What may not be intuitive is why memory usage continues to grow after the first iteration. After all, it built a big array, but then promptly discarded it and started building a new one of the same size. Can’t it just use the space freed up by the previous array? The answer, which explains our problem, is no. Aside from tuning the garbage collector, you don’t have control over when it runs, and what we’re seeing in the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">build_arrays.rb</code> example is new memory allocations being made prior to garbage collection of our old, discarded objects.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
I should point out that this isn’t some sort of horrible memory management issue specific to <a href="https://www.toptal.com/ruby" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ruby</a>, but is generally applicable to garbage-collected languages. Just to reassure myself of this, I reproduced essentially the same example with Go and saw similar results. However, there are Ruby libraries that make it easy to create this sort of memory issue.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
So if we need to work with large chunks of data, are we doomed to just throw lots of RAM at our problem? Thankfully, that’s not the case. If we take the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">build_arrays.rb</code> example and decrease the array size, we’ll see a decrease in the point where memory usage plateaus that’s roughly proportional to the array size.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92735/toptal-blog-image-1467709209945-e542e3b4cbf40d63d94f48368a3e2252.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
This means that if we can break our work into smaller pieces to process and avoid having too many objects existing at one time, we can dramatically reduce the memory footprint. Unfortunately, that often means taking nice, clean code and turning it into more code that does the same thing, just in a more memory-efficient way.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
In a real codebase, the source of a memory issue will likely not be as obvious as in the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">build_arrays.rb</code>example. Isolating a memory issue before trying to actually dig in and fix it is essential because it’s easy to make incorrect assumptions about what’s causing the problem.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92742/toptal-blog-image-1467724640253-e82ed4bd6ccf9cc23b975c75f236346b.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
I generally use two approaches, often in combination, to track down memory issues: leaving the code intact and wrapping a profiler around it, and monitoring memory usage of the process while disabling/enabling different parts of the code I suspect could be problematic. I’ll be using <a href="https://github.com/SamSaffron/memory_profiler" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">memory_profiler</a> here for profiling, but <a href="https://github.com/ruby-prof/ruby-prof" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ruby-prof</a> is another popular option, and <a href="https://github.com/schneems/derailed_benchmarks" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">derailed_benchmarks</a> has some great Rails-specific capabilities.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Here’s some code that’ll use a bunch of memory, where it may not be immediately clear which step is pushing up memory usage the most:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Using <a href="https://github.com/schneems/get_process_mem" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">get_process_mem</a>, we can quickly verify that it does use a lot of memory when there are a lot of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Person</code> records being created.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Result:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Looking through the code, there are multiple steps that seem like good candidates for using a lot of memory: building a big array of strings, calling <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">#to_a</code> on an Active Record relation to make a big array of Active Record objects (not a great idea, but done for demonstration purposes), and serializing the array of Active Record objects.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
We can then profile this code to see where memory allocations are happening:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Note that the number being fed to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">run</code> here is 1/10 of the previous example, since the profiler itself uses a lot of memory, and can actually lead to memory exhaustion when profiling code that already causes high memory usage.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
The results file is rather lengthy and includes memory and object allocation and retention at the gem, file, and location levels. There’s a wealth of information to explore, but here are a couple of interesting snippets:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
We see the most allocations happening inside Active Record, which would seem to point at either instantiating all the objects in the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">records</code> array, or serialization with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">#to_json</code>. Next, we can test our memory usage without the profiler while disabling these suspects. We can’t disable retrieving <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">records</code> and still be able to do the serialization step, so let’s try disabling serialization first.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Result:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
That does indeed seem to be where most of the memory is going, with before/after memory delta dropping 81% by skipping it. We can also see what happens if we stop forcing the big array of records to be created.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Result:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
This reduces memory usage as well, although it’s an order of magnitude less reduction than disabling serialization. So at this point, we know our biggest culprits, and can make a decision about what to optimize based on this data.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Although the example here was contrived, the approaches are generally applicable. Profiler results may not point you at the exact spot in your code where the problem lies, and can also be misinterpreted, so it’s a good idea to follow up by looking at actual memory usage while turning sections of code on and off. Next, we’ll look at some common cases where memory usage becomes an issue and how to optimize them.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
A common source of memory issues is deserializing large amounts of data from XML, JSON or some other data serialization format. Using methods like <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">JSON.parse</code> or Active Support’s <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Hash.from_xml</code> is incredibly convenient, but when the data you’re loading is large, the resulting data structure that’s loaded in memory can be enormous.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
If you have control over the source of the data, you can do things to limit the amount of data you’re receiving, like adding filtering or pagination support. But if it’s an external source or one you can’t control, another option is to use a streaming deserializer. For XML, <a href="https://github.com/ohler55/ox" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Ox</a> is one option, and for JSON <a href="https://github.com/brianmario/yajl-ruby" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">yajl-ruby</a> appears to operate similarly, although I don’t have much experience with it.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Here’s an example of parsing a 1.7MB XML file, using <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Hash#from_xml</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
111MB for a 1.7MB file! This clearly is not going to scale up well. Here’s the streaming parser version.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
This brings us down to a negligible memory increase and should be able to handle vastly larger files. However, the tradeoff is that we now have 28 lines of handler code we didn’t need before, which seems like it’d be error prone, and for production use it should have some tests around it.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As we saw in the section about isolating memory usage hotspots, serialization can have high memory costs. Here’s the key part of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">people.rb</code> from earlier.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Running this with 100,000 records in the database, we get:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
The issue with calling <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">#to_json</code> here is that it instantiates an object for every record, and then encodes to JSON. Generating the JSON record-by-record so that only one record object would need to exist at a time reduces the memory usage significantly. None of the popular Ruby JSON libraries appear to handle this, but a commonly recommended approach is to build the JSON string manually. There is a <a href="https://github.com/camertron/json-write-stream" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">json-write-stream</a> gem that provides a nice API for doing this, and converting our example to this looks like:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Once again, we see optimization has given us more code, but the result seems worth it:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
A great feature added to Ruby starting with 2.0 is the ability to make enumerators lazy. This is great for improving memory usage when chaining methods on an enumerator. Let’s start with some code that isn’t lazy:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Result:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92741/toptal-blog-image-1467723471018-4e2536a4ee1d8f9545c926f3b88a99b4.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
What happens here is that at each step in the chain, it iterates over every element in the enumerator, producing an array that has the subsequent method in the chain invoked on it, and so forth. Let’s see what happens when we make this lazy, which just requires adding a call to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">lazy</code> on the enumerator we get from <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">times</code>:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Result:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Finally, an example that gives us a huge memory usage win, without adding a lot of extra code! Note that if we didn’t need to accumulate any results at the end, for instance, if each item was saved to the database and could then be forgotten, there would be even less memory usage. To make a lazy enumerable evaluate at the end of the chain, just add a final call to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">force</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Another thing to note about the example is that the chain starts with a call to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">times</code> prior to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">lazy</code>, which uses very little memory since it just returns an enumerator that will generate an integer each time it’s invoked. So if an enumerable can be used instead of a big array at the beginning of the chain, that will help.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
One real-world application of building an enumerable to lazily feed into some sort of processing pipeline is processing paginated data. So rather than requesting all the pages and putting them into one big array, they could be exposed through an enumerator that nicely hides all the pagination details. This could look something like:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
We’ve done some characterization of memory usage in Ruby, and looked at some general tools for tracking down memory issues, as well as some common cases and ways to improve them. The common cases we explored are by no means comprehensive and are highly biased by the sort of issues I personally have encountered. However, the biggest gain may just be getting in the mindset of thinking about how the code will impact memory usage.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
<i>This post originally appeared in <a href="https://www.toptal.com/ruby/hunting-ruby-memory-issues" target="_blank">Toptal Engineering</a> blog</i></div>
Anonymousnoreply@blogger.com3tag:blogger.com,1999:blog-8706416286757464743.post-76463755321700492642016-07-14T16:36:00.000+07:002016-07-14T16:36:21.385+07:00Scaling Scala: How to Dockerize Using Kubernetes<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://kubernetes.io/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Kubernetes</a> is the new kid on the block, promising to help deploy applications into the cloud and scale them more quickly. Today, when developing for a microservices architecture, it’s pretty standard to choose Scala for creating API servers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">Microservices are replacing classic monolithic back-end servers with multiple independent services that communicate among themselves and have their own processes and resources.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If there is a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Scala</em> application in your plans and you want to <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">scale</em> it into a cloud, then you are at the right place. In this article I am going to show step-by-step how to take a generic Scala application and implement Kubernetes with Docker to launch multiple instances of the application. The final result will be a single application deployed as multiple instances, and load balanced by Kubernetes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All of this will be implemented by simply importing the <a href="https://github.com/sciabarra/ScalaGoodies/tree/kubernetes" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Kubernetes source kit</a> in your Scala application. Please note, the kit hides a lot of complicated details related to installation and configuration, but it is small enough to be readable and easy to understand if you want to analyze what it does. For simplicity, we will deploy everything on your local machine. However, the same configuration is suitable for a real-world cloud deployment of Kubernetes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Scale Your Scala Application with Kubernetes" src="https://assets.toptal.io/uploads/blog/image/92679/toptal-blog-image-1466614318248-8d9deae244549b8191c13a84c070cdc6.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Be smart and sleep tight, scale your Docker with Kubernetes.</span></div>
<h2 id="what-is-kubernetes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What is Kubernetes?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Before going into the gory details of the implementation, let’s discuss what Kubernetes is and why it’s important.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You may have already heard of Docker. In a sense, it is a lightweight virtual machine.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">Docker gives the advantage of deploying each server in an isolated environment, very similar to a stand-alone virtual machine, without the complexity of managing a full-fledged virtual machine.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For these reasons, it is already one of the more widely used tools for deploying applications in clouds. A Docker image is pretty easy and fast to build and duplicable, much easier than a traditional virtual machine like VMWare, VirtualBox, or XEN.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Kubernetes complements Docker, offering a complete environment for managing dockerized applications. By using Kubernetes, you can easily deploy, configure, orchestrate, manage, and monitor hundreds or even thousands of Docker applications.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Kubernetes is an open source tool developed by Google and has been adopted by many other vendors. Kubernetes is available natively on the Google cloud platform, but other vendors have adopted it for their OpenShift cloud services too. It can be found on Amazon AWS, Microsoft Azure, RedHat OpenShift, and even more cloud technologies. We can say it is well positioned to become a standard for deploying cloud applications.</div>
<h2 id="prerequisites" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Prerequisites</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we covered the basics, let’s check if you have all the prerequisite software installed. First of all, you need <a href="http://www.docker.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Docker</a>. If you are using either Windows or Mac, you need the <a href="https://www.docker.com/products/docker-toolbox" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Docker Toolbox</a>. If you are using Linux, you need to install the particular package provided by your distribution or simply <a href="https://docs.docker.com/linux/step_one/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">follow the official directions</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We are going to code in Scala, which is a JVM language. You need, of course, the Java Development Kit and the <a href="http://www.scala-sbt.org/download.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">scala SBT tool</a> installed and available in the global path. If you are already a Scala programmer, chances are you have those tools already installed.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you are using Windows or Mac, Docker will by default create a virtual machine named <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">default</code> with only 1GB of memory, which can be too small for running Kubernetes. In my experience, I had issues with the default settings. I recommend that you open the VirtualBox GUI, select your virtual machine <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">default</code>, and change the memory to at least to 2048MB.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="VirtualBox memory settings" src="https://assets.toptal.io/uploads/blog/image/92652/toptal-blog-image-1466164786062-12c76d6bf380690d154c95102b093f09.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="the-application-to-clusterize" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Application to Clusterize</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The instructions in this tutorial can apply to any Scala application or project. For this article to have some “meat” to work on, I chose an example used very often to demonstrate a simple REST microservice in Scala, called <a href="https://github.com/theiterators/akka-http-microservice" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Akka HTTP</a>. I recommend you try to apply source kit to the suggested example before attempting to use it on your application. I have tested the kit against the demo application, but I cannot guarantee that there will be no conflicts with your code.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So first, we start by cloning the demo application:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">git clone https://github.com/theiterators/akka-http-microservice
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Next, test if everything works correctly:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">cd</span> akka-http-microservice
sbt run
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Then, access to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">http://localhost:9000/ip/8.8.8.8</code>, and you should see something like in the following image:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Akka HTTP microservice is running" src="https://assets.toptal.io/uploads/blog/image/92653/toptal-blog-image-1466164825919-d5da64440f8bdef474def9125f2e6721.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="adding-the-source-kit" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Adding the Source Kit</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, we can add the source kit with some Git magic:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">git remote add ScalaGoodies https://github.com/sciabarra/ScalaGoodies
git fetch --all
git merge ScalaGoodies/kubernetes
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With that, you have the demo including the source kit, and you are ready to try. Or you can even copy and paste the code from there into your application.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once you have merged or copied the files in your projects, you are ready to start.</div>
<h2 id="starting-kubernetes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Starting Kubernetes</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once you have downloaded the kit, we need to download the necessary <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">kubectl</code> binary, by running:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/install.sh
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This installer is smart enough (hopefully) to download the correct <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">kubectl</code> binary for OSX, Linux, or Windows, depending on your system. Note, the installer worked on the systems I own. Please do report any issues, so that I can fix the kit.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once you have installed the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">kubectl</code> binary, you can start the whole Kubernetes in your local Docker. Just run:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/start-local-kube.sh
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first time it is run, this command will download the images of the whole Kubernetes stack, and a local registry needed to store your images. It can take some time, so please be patient. Also note, it needs direct accesses to the internet. If you are behind a proxy, it will be a problem as the kit does not support proxies. To solve it, you have to configure the tools like Docker, curl, and so on to use the proxy. It is complicated enough that I recommend getting a temporary unrestricted access.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Assuming you were able to download everything successfully, to check if Kubernetes is running fine, you can type the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-bash hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl get nodes
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The expected answer is:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">NAME STATUS AGE
127.0.0.1 Ready 2m
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that age may vary, of course. Also, since starting Kubernetes can take some time, you may have to invoke the command a couple of times before you see the answer. If you do not get errors here, congratulations, you have Kubernetes up and running on your local machine.</div>
<h2 id="dockerizing-your-scala-app" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Dockerizing Your Scala App</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that you have Kubernetes up and running, you can deploy your application in it. In the old days, before Docker, you had to deploy an entire server for running your application. With Kubernetes, all you need to do to deploy your application is:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Create a Docker image.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Push it in a registry from where it can be launched.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Launch the instance with Kubernetes, that will take the image from the registry.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Luckily, it is way less complicated that it looks, especially if you are using the SBT build tool like many do.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the kit, I included two files containing all the necessary definitions to create an image able to run Scala applications, or at least what is needed to run the Akka HTTP demo. I cannot guarantee that it will work with any other Scala applications, but it is a good starting point, and should work for many different configurations. The files to look for building the Docker image are:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">docker.sbt
project/docker.sbt
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s have a look at what’s in them. The file <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">project/docker.sbt</code> contains the command to import the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sbt-docker</code> plugin:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">addSbtPlugin("se.marcuslonnberg" % "sbt-docker" % "1.4.0")
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This plugin manages the building of the Docker image with SBT for you. The Docker definition is in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">docker.sbt</code> file and looks like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">imageNames in docker := Seq(ImageName("localhost:5000/akkahttp:latest"))
dockerfile in docker := {
val jarFile: File = sbt.Keys.`package`.in(Compile, packageBin).value
val classpath = (managedClasspath in Compile).value
val mainclass = mainClass.in(Compile, packageBin).value.getOrElse(sys.error("Expected exactly one main class"))
val jarTarget = s"/app/${jarFile.getName}"
val classpathString = classpath.files.map("/app/" + _.getName)
.mkString(":") + ":" + jarTarget
new Dockerfile {
from("anapsix/alpine-java:8")
add(classpath.files, "/app/")
add(jarFile, jarTarget)
entryPoint("java", "-cp", classpathString, mainclass)
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To fully understand the meaning of this file, you need to know Docker well enough to understand this definition file. However, we are not going into the details of the Docker definition file, because you do not need to understand it thoroughly to build the image.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">The beauty of using SBT for building the Docker image is that </span><span style="background-color: #fafafa; color: #505050; font-size: 1.1em; font-style: italic; line-height: 1.5em; text-align: center;">the SBT will take care of collecting all the files for you.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note the classpath is automatically generated by the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">val classpath = (managedClasspath in Compile).value
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In general, it is pretty complicated to gather all the JAR files to run an application. Using SBT, the Docker file will be generated with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">add(classpath.files, "/app/")</code>. This way, SBT collects all the JAR files for you and constructs a Dockerfile to run your application.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The other commands gather the missing pieces to create a Docker image. The image will be built using an existing image APT to run Java programs (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">anapsix/alpine-java:8</code>, available on the internet in the Docker Hub). Other instructions are adding the other files to run your application. Finally, by specifying an entry point, we can run it. Note also that the name starts with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">localhost:5000</code> on purpose, because <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">localhost:5000</code> is where I installed the registry in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">start-kube-local.sh</code> script.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="building-the-docker-image-with-sbt" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Building the Docker Image with SBT</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To build the Docker image, you can ignore all the details of the Dockerfile. You just need to type:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">sbt dockerBuildAndPush
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sbt-docker</code> plugin will then build a Docker image for you, downloading from the internet all the necessary pieces, and then it will push to a Docker registry that was started before, together with the Kubernetes application in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">localhost</code>. So, all you need is to wait a little bit more to have your image cooked and ready.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note, if you experience problems, the best thing to do is to reset everything to a known state by running the following commands:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/stop-kube-local.sh
bin/start-kube-local.sh
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Those commands should stop all the containers and restart them correctly to get your registry ready to receive the image built and pushed by <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sbt</code>.</div>
<h2 id="starting-the-service-in-kubernetes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Starting the Service in Kubernetes</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that the application is packaged in a container and pushed in a registry, we are ready to use it. Kubernetes uses the command line and configuration files to manage the cluster. Since command lines can become very long, and also be able to replicate the steps, I am using the configurations files here. All the samples in the source kit are in the folder <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">kube</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Our next step is to launch a single instance of the image. A running image is called, in the Kubernetes language, a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">pod</em>. So let’s create a pod by invoking the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl create -f kube/akkahttp-pod.yml
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can now inspect the situation with the command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl get pods
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should see:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">NAME READY STATUS RESTARTS AGE
akkahttp 1/1 Running 0 33s
k8s-etcd-127.0.0.1 1/1 Running 0 7d
k8s-master-127.0.0.1 4/4 Running 0 7d
k8s-proxy-127.0.0.1 1/1 Running 0 7d
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Status actually can be different, for example, “ContainerCreating”, it can take a few seconds before it becomes “Running”. Also, you can get another status like “Error” if, for example, you forget to create the image before.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can also check if your pod is running with the command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl logs akkahttp
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should see an output ending with something like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">[DEBUG] [05/30/2016 12:19:53.133] [default-akka.actor.default-dispatcher-5] [akka://default/system/IO-TCP/selectors/$a/0] Successfully bound to /0:0:0:0:0:0:0:0:9000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now you have the service up and running inside the container. However, the service is not yet reachable. This behavior is part of the design of Kubernetes. Your pod is running, but you have to expose it explicitly. Otherwise, the service is meant to be internal.</div>
<h2 id="creating-a-service" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Creating a Service</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Creating a service and checking the result is a matter of executing:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl create -f kube/akkahttp-service.yaml
bin/kubectl get svc
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should see something like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
akkahttp-service 10.0.0.54 9000/TCP 44s
kubernetes 10.0.0.1 <none> 443/TCP 3m
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that the port can be different. Kubernetes allocated a port for the service and started it. If you are using Linux, you can directly open the browser and type <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">http://10.0.0.54:9000/ip/8.8.8.8</code> to see the result. If you are using Windows or Mac with Docker Toolbox, the IP is local to the virtual machine that is running Docker, and unfortunately it is still unreachable.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I want to stress here that this is not a problem of Kubernetes, rather it is a limitation of the Docker Toolbox, which in turn depends on the constraints imposed by virtual machines like VirtualBox, which act like a computer within another computer. To overcome this limitation, we need to create a tunnel. To make things easier, I included another script which opens a tunnel on an arbitrary port to reach any service we deployed. You can type the following command:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/forward-kube-local.sh akkahttp-service 9000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that the tunnel will not run in the background, you have to keep the terminal window open as long as you need it and close when you do not need the tunnel anymore. While the tunnel is running, you can open: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">http://localhost:9000/ip/8.8.8.8</code> and finally see the application running in Kubernetes.</div>
<h2 id="final-touch-scale" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Final Touch: Scale</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So far we have “simply” put our application in Kubernetes. While it is an exciting achievement, it does not add too much value to our deployment. We’re saved from the effort of uploading and installing on a server and configuring a proxy server for it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Where Kubernetes shines is in scaling. You can deploy two, ten, or one hundred instances of our application by only changing the number of replicas in the configuration file. So let’s do it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We are going to stop the single pod and start a deployment instead. So let’s execute the following commands:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl delete -f kube/akkahttp-pod.yml
bin/kubectl create -f kube/akkahttp-deploy.yaml
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Next, check the status. Again, you may try a couple of times because the deployment can take some time to be performed:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">NAME READY STATUS RESTARTS AGE
akkahttp-deployment-4229989632-mjp6u 1/1 Running 0 16s
akkahttp-deployment-4229989632-s822x 1/1 Running 0 16s
k8s-etcd-127.0.0.1 1/1 Running 0 6d
k8s-master-127.0.0.1 4/4 Running 0 6d
k8s-proxy-127.0.0.1 1/1 Running 0 6d
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now we have two pods, not one. This is because in the configuration file I provided, there is the value <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">replica: 2</code>, with two different names generated by the system. I am not going into the details of the configuration files, because the scope of the article is simply an introduction for Scala programmers to jump-start into Kubernetes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Anyhow, there are now two pods active. What is interesting is that the service is the same as before. We configured the service to load balance between all the pods labeled <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">akkahttp</code>. This means we do not have to redeploy the service, but we can replace the single instance with a replicated one.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can verify this by launching the proxy again (if you are on Windows and you have closed it):</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/forward-kube-local.sh akkahttp-service 9000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Then, we can try to open two terminal windows and see the logs for each pod. For example, in the first type:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl logs -f akkahttp-deployment-4229989632-mjp6u
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And in the second type:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">bin/kubectl logs -f akkahttp-deployment-4229989632-s822x
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Of course, edit the command line accordingly with the values you have in your system.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, try to access the service with two different browsers. You should expect to see the requests to be split between the multiple available servers, like in the following image:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Kubernets in action" src="https://assets.toptal.io/uploads/blog/image/92673/toptal-blog-image-1466434011664-3646b46213ffc09ab879f4d5b794cc8c.png" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 999.5px;" /></div>
<h2 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Conclusion</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Today we barely scratched the surface. Kubernetes offers a lot more possibilities, including automated scaling and restart, incremental deployments, and volumes. Furthermore, the application we used as an example is very simple, stateless with the various instances not needing to know each other. In the real world, distributed applications <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</em> need to know each other, and need to change configurations according to the availability of other servers. Indeed, Kubernetes offers a distributed keystore (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">etcd</code>) to allow different applications to communicate with each other when new instances are deployed. However, this example is purposefully small enough and simplified to help you get going, focusing on the core functionalities. If you follow the tutorial, you should be able to get a working environment for your <a href="https://www.toptal.com/scala" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Scala</a> application on your machine without being confused by a large number of details and getting lost in the complexity.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/michele-sciabarra" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Michele Sciabarra</a>, a <a href="https://www.toptal.com/scala/scale-your-scala-application-with-kubernetes">Toptal</a><a href="https://www.toptal.com/scala"> Scala developer</a>.</div>
Anonymousnoreply@blogger.com4tag:blogger.com,1999:blog-8706416286757464743.post-43392930928402890922016-06-28T12:55:00.000+07:002016-06-28T12:55:27.170+07:00Boost Your Data Munging with R<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The R language is often perceived as a language for statisticians and data scientists. Quite a long time ago, this was mostly true. However, over the years the flexibility R provides via packages has made R into a more general purpose language. R was open sourced in 1995, and since that time <a href="http://www.techrepublic.com/article/exponential-growth-of-rs-open-source-community-threatens-commercial-competitors/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">repositories of R packages are constantly growing</a>. Still, compared to languages like Python, R is strongly based around the data.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Speaking about data, tabular data deserves particular attention, as it’s one of the most commonly used data types. It is a data type which corresponds to a table structure known in databases, where each column can be of a different type, and processing performance of that particular data type is the crucial factor for many applications.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="R can be used for very efficient data munging of tabular data" src="https://assets.toptal.io/uploads/blog/image/92640/toptal-blog-image-1465810970332-5b821f82d3327bc01d1c1617f4587915.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">R can be used for very efficient data munging of tabular data</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this article, we are going to present how to achieve tabular data transformation in an efficient manner. Many<a href="https://www.toptal.com/r" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">people who use R</a> already for machine learning are not aware that <a href="https://en.wikipedia.org/wiki/Data_wrangling" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">data munging</a> can be done faster in R, and that they do not need to use another tool for it.</div>
<h2 id="high-performance-solution-in-r" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
High-performance Solution in R</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Base R introduced the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code> class in the year 1997, which was based on S-PLUS before it. Unlike commonly used databases which store data row by row, R <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code> stores the data in memory as a column-oriented structure, thus making it more cache-efficient for column operations which are common in analytics. Additionally, even though R is a functional programming language, it does not enforce that on the developer. Both opportunities have been well addressed by <a href="https://github.com/Rdatatable/data.table/wiki" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code></a> R package, which is available in CRAN repository. It performs quite fast when grouping operations, and is particularly memory efficient by being careful about materializing intermediate data subsets, such as materializing only those columns necessary for a certain task. It also avoids unnecessary copies through its <a href="https://rawgit.com/wiki/Rdatatable/data.table/vignettes/datatable-reference-semantics.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">reference semantics</a> while adding or updating columns. The first version of the package has been published in April 2006, significantly improving <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code> performance at that time. The initial package description was:</div>
<blockquote style="background-color: white; border-left-color: rgb(240, 240, 240); border-left-style: solid; border-width: 0px 0px 0px 10px; box-sizing: border-box; color: #505050; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px 0px 0px 15px; quotes: none; vertical-align: baseline;">
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This package does very little. The only reason for its existence is that the white book specifies that data.frame must have rownames. This package defines a new class data.table which operates just like a data.frame, but uses up to 10 times less memory, and can be up to 10 times faster to create (and copy). It also takes the opportunity to allow subset() and with() like expressions inside the []. Most of the code is copied from base functions with the code manipulating row.names removed.</div>
</blockquote>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since then, both <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> implementations have been improved, but <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> remains to be incredibly faster than base R. In fact, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> isn’t just faster than base R, but it appears to be one of the fastest open-source data wrangling tool available, competing with tools like <a href="https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Python Pandas</a>, and columnar storage databases or big data apps like <a href="https://github.com/szilard/benchm-databases" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Spark</a>. Its performance over distributed shared infrastructure hasn’t been yet benchmarked, but being able to have up to two billion rows on a single instance gives promising prospects. Outstanding performance goes hand-in-hand with the <a href="https://stackoverflow.com/questions/21435339/data-table-vs-dplyr-can-one-do-something-well-the-other-cant-or-does-poorly" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">functionalities</a>. Additionally, with recent efforts at parallelizing time-consuming parts for incremental performance gains, one direction towards pushing the performance limit seems quite clear.</div>
<h2 id="data-transformation-examples" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Data Transformation Examples</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Learning R gets a little bit easier because of the fact that it works interactively, so we can follow examples step by step and look at the results of each step at any time. Before we start, let’s install the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code>package from CRAN repository.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">install.packages(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"data.table"</span>)
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Useful hint</span>: We can open the manual of any function just by typing its name with leading question mark, i.e. <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">?install.packages</code>.</div>
<h3 id="loading-data-into-r" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Loading Data into R</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are tons of packages for extracting data from a wide range of formats and databases, which often includes native drivers. We will load data from the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">CSV</em> file, the most common format for raw tabular data. File used in the following examples can be found <a href="https://github.com/Rdatatable/data.table/blob/eba8eeb7bad2d46007bcde8d640d0fae79b9939d/vignettes/flights14.csv" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">here</a>. We don’t have to bother about <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CSV</code> reading performance as the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">fread</code> function is highly optimized on that.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to use any function from a package, we need to load it with the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">library</code> call.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">library</span>(data.table)
DT <- fread(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"flights14.csv"</span>)
print(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## year month day dep_delay arr_delay carrier origin dest air_time
## 1: 2014 1 1 14 13 AA JFK LAX 359
## 2: 2014 1 1 -3 13 AA JFK LAX 363
## 3: 2014 1 1 2 9 AA JFK LAX 351
## 4: 2014 1 1 -8 -26 AA LGA PBI 157
## 5: 2014 1 1 2 1 AA JFK LAX 350
## ---
## 253312: 2014 10 31 1 -30 UA LGA IAH 201
## 253313: 2014 10 31 -5 -14 UA EWR IAH 189
## 253314: 2014 10 31 -8 16 MQ LGA RDU 83
## 253315: 2014 10 31 -4 15 MQ LGA DTW 75
## 253316: 2014 10 31 -5 1 MQ LGA SDF 110
## distance hour
## 1: 2475 9
## 2: 2475 11
## 3: 2475 19
## 4: 1035 7
## 5: 2475 13
## ---
## 253312: 1416 14
## 253313: 1400 8
## 253314: 431 11
## 253315: 502 11
## 253316: 659 8
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If our data is not well modeled for further processing, as they need to be reshaped from long-to-wide or wide-to-long (also known as <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">pivot</em> and <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">unpivot</em>) format, we may look at <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">?dcast</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">?melt</code> functions, known from<a href="https://github.com/hadley/reshape" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">reshape2</a> package. However, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> implements faster and memory efficient methods for data.table/data.frame class.</div>
<h3 id="querying-with-datatable-syntax" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Querying with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> Syntax</h3>
<h4 id="if-youre-familiar-with-dataframe" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If You’re Familiar with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code></h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> is very similar to query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code>. While filtering in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">i</code> argument, we can use column names directly without the need to access them with the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">$</code> sign, like <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">df[df$col > 1, ]</code>. When providing the next argument <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code>, we provide an expression to be evaluated in the scope of our <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code>. To pass a non-expression <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code> argument use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">with=FALSE</code>. Third argument, not present in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code> method, defines the groups, making the expression in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code> to be evaluated by groups.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># data.frame</span>
DF[DF$col1 > <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1L</span>, c(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"col2"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"col3"</span>)]
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># data.table</span>
DT[col1 > <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1L</span>, .(col2, col3), <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">...</span>] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># by group using: `by = col4`</span>
</code></pre>
<h4 id="if-youre-familiar-with-databases" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If You’re Familiar with Databases</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> in many aspects corresponds to SQL queries that more people might be familiar with. <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">DT</code>below represents <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> object and corresponds to SQLs <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">FROM</code> clause.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[ i = where,
j = select | update,
by = group by]
[ having, ... ]
[ order by, ... ]
[ ... ] ... [ ... ]
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Untangling Tabluar Data" src="https://assets.toptal.io/uploads/blog/image/92639/toptal-blog-image-1465810897187-32008e617dce77f9fd78acb5ac2791bb.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h3 id="sorting-rows-and-re-ordering-columns" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sorting Rows and Re-Ordering Columns</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sorting data is a crucial transformation for time series, and it is also imports for data extract and presentation. Sort can be achieved by providing the integer vector of row order to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">i</code> argument, the same way as <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code>. First argument in query <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">order(carrier, -dep_delay)</code> will select data in ascending order on <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">carrier</code> field and descending order on <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">dep_delay</code> measure. Second argument <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code>, as described in the previous section, defines the columns (or expressions) to be returned and their order.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ans <- DT[order(carrier, -dep_delay),
.(carrier, origin, dest, dep_delay)]
head(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier origin dest dep_delay
## 1: AA EWR DFW 1498
## 2: AA JFK BOS 1241
## 3: AA EWR DFW 1071
## 4: AA EWR DFW 1056
## 5: AA EWR DFW 1022
## 6: AA EWR DFW 989
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To re-order data by reference, instead of querying data in specific order, we use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">set*</code> functions.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">setorder(DT, carrier, -dep_delay)
leading.cols <- c(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"carrier"</span>,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dep_delay"</span>)
setcolorder(DT, c(leading.cols, setdiff(names(DT), leading.cols)))
print(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1241 2014 4 15 1223 JFK BOS 39
## 3: AA 1071 2014 6 13 1064 EWR DFW 175
## 4: AA 1056 2014 9 12 1115 EWR DFW 198
## 5: AA 1022 2014 6 16 1073 EWR DFW 178
## ---
## 253312: WN -12 2014 3 9 -21 LGA BNA 115
## 253313: WN -13 2014 3 10 -18 EWR MDW 112
## 253314: WN -13 2014 5 17 -30 LGA HOU 202
## 253315: WN -13 2014 6 15 10 LGA MKE 101
## 253316: WN -13 2014 8 19 -30 LGA CAK 63
## distance hour
## 1: 1372 7
## 2: 187 13
## 3: 1372 10
## 4: 1372 6
## 5: 1372 7
## ---
## 253312: 764 16
## 253313: 711 20
## 253314: 1428 17
## 253315: 738 20
## 253316: 397 16
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Most often, we don’t need <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">both</em> the original dataset and the ordered/sorted dataset. By default, the R language, similar to other functional programming languages, will return sorted data as new object, and thus will require twice as much memory as sorting by reference.</div>
<h3 id="subset-queries" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Subset Queries</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s create a subset dataset for flight origin “JFK” and month from 6 to 9. In the second argument, we subset results to listed columns, adding one calculated variable <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sum_delay</code>.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ans <- DT[origin == <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"JFK"</span> & month %<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span>% <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">6</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">9</span>,
.(origin, month, arr_delay, dep_delay, sum_delay = arr_delay + dep_delay)]
head(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## origin month arr_delay dep_delay sum_delay
## 1: JFK 7 925 926 1851
## 2: JFK 8 727 772 1499
## 3: JFK 6 466 451 917
## 4: JFK 7 414 450 864
## 5: JFK 6 411 442 853
## 6: JFK 6 333 343 676
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By default, when subsetting dataset on single column <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> will automatically create an index for that column. This results in <a href="https://jangorecki.github.io/blog/2015-11-23/data.table-index.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">real-time</a> answers on any further filtering calls on that column.</div>
<h3 id="update-dataset" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Update Dataset</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Adding a new column by reference is performed using the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">:=</code> operator, it assigns a variable into dataset in place. This avoids in-memory copy of dataset, so we don’t need to assign results to each new variable.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[, sum_delay := arr_delay + dep_delay]
head(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1241 2014 4 15 1223 JFK BOS 39
## 3: AA 1071 2014 6 13 1064 EWR DFW 175
## 4: AA 1056 2014 9 12 1115 EWR DFW 198
## 5: AA 1022 2014 6 16 1073 EWR DFW 178
## 6: AA 989 2014 6 11 991 EWR DFW 194
## distance hour sum_delay
## 1: 1372 7 2992
## 2: 187 13 2464
## 3: 1372 10 2135
## 4: 1372 6 2171
## 5: 1372 7 2095
## 6: 1372 11 1980
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To add more variables at once, we can use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">DT[,</code>:=<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">(sum_delay = arr_delay + dep_delay)]</code> syntax, similar to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.(sum_delay = arr_delay + dep_delay)</code> when querying from dataset.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It is possible to sub-assign by reference, updating only particular rows in place, just by combining with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">i</code>argument.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[origin==<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"JFK"</span>,
distance := <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">NA</span>]
head(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1241 2014 4 15 1223 JFK BOS 39
## 3: AA 1071 2014 6 13 1064 EWR DFW 175
## 4: AA 1056 2014 9 12 1115 EWR DFW 198
## 5: AA 1022 2014 6 16 1073 EWR DFW 178
## 6: AA 989 2014 6 11 991 EWR DFW 194
## distance hour sum_delay
## 1: 1372 7 2992
## 2: NA 13 2464
## 3: 1372 10 2135
## 4: 1372 6 2171
## 5: 1372 7 2095
## 6: 1372 11 1980</code></pre>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><br /><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="aggregate-data" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Aggregate Data</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To aggregate data, we provide the third argument <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">by</code> to the square bracket. Then, in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code> we need to provide aggregate function calls, so the data can be actually aggregated. The <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.N</code> symbol used in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code> argument corresponds to the number of all observations in each group. As previously mentioned, aggregates can be combined with subsets on rows and selecting columns.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ans <- DT[,
.(m_arr_delay = mean(arr_delay),
m_dep_delay = mean(dep_delay),
count = .N),
.(carrier, month)]
head(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier month m_arr_delay m_dep_delay count
## 1: AA 10 5.541959 7.591497 2705
## 2: AA 4 1.903324 3.987008 2617
## 3: AA 6 8.690067 11.476475 2678
## 4: AA 9 -1.235160 3.307078 2628
## 5: AA 8 4.027474 8.914054 2839
## 6: AA 7 9.159886 11.665953 2802
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Often, we may need to compare a value of a row to its aggregate over a group. In SQL, we apply <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">aggregates over partition by</em>: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">AVG(arr_delay) OVER (PARTITION BY carrier, month)</code>.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ans <- DT[,
.(arr_delay, carrierm_mean_arr = mean(arr_delay),
dep_delay, carrierm_mean_dep = mean(dep_delay)),
.(carrier, month)]
head(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier month arr_delay carrierm_mean_arr dep_delay carrierm_mean_dep
## 1: AA 10 1494 5.541959 1498 7.591497
## 2: AA 10 840 5.541959 848 7.591497
## 3: AA 10 317 5.541959 338 7.591497
## 4: AA 10 292 5.541959 331 7.591497
## 5: AA 10 322 5.541959 304 7.591497
## 6: AA 10 306 5.541959 299 7.591497
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we don’t want to query data with those aggregates, and instead just put them into actual table updating by reference, we can accomplish that with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">:=</code> operator. This avoids the in-memory copy of the dataset, so we don’t need to assign results to the new variable.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[,
`:=`(carrierm_mean_arr = mean(arr_delay),
carrierm_mean_dep = mean(dep_delay)),
.(carrier, month)]
head(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1241 2014 4 15 1223 JFK BOS 39
## 3: AA 1071 2014 6 13 1064 EWR DFW 175
## 4: AA 1056 2014 9 12 1115 EWR DFW 198
## 5: AA 1022 2014 6 16 1073 EWR DFW 178
## 6: AA 989 2014 6 11 991 EWR DFW 194
## distance hour sum_delay carrierm_mean_arr carrierm_mean_dep
## 1: 1372 7 2992 5.541959 7.591497
## 2: NA 13 2464 1.903324 3.987008
## 3: 1372 10 2135 8.690067 11.476475
## 4: 1372 6 2171 -1.235160 3.307078
## 5: 1372 7 2095 8.690067 11.476475
## 6: 1372 11 1980 8.690067 11.476475
</code></pre>
<h3 id="join-datasets" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Join Datasets</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Base R joining and merging of datasets is considered a special type of <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">subset</em> operation. We provide a dataset to which we want to join in the first square bracket argument <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">i</code>. For each row in dataset provided to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">i</code>, we match rows from the dataset in which we use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">[</code>. If we want to keep only matching rows (<em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">inner join</em>), then we pass an extra argument <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">nomatch = 0L</code>. We use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">on</code> argument to specify columns on which we want to join both datasets.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># create reference subset</span>
carrierdest <- DT[, .(count=.N), .(carrier, dest) <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># count by carrier and dest</span>
][<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># just 10 first groups</span>
] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># chaining `[...][...]` as subqueries</span>
print(carrierdest)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dest count
## 1: AA DFW 5877
## 2: AA BOS 1173
## 3: AA ORD 4798
## 4: AA SEA 298
## 5: AA EGE 85
## 6: AA LAX 3449
## 7: AA MIA 6058
## 8: AA SFO 1312
## 9: AA AUS 297
## 10: AA DCA 172
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># outer join</span>
ans <- carrierdest[DT, on = c(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"carrier"</span>,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dest"</span>)]
print(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dest count dep_delay year month day arr_delay origin
## 1: AA DFW 5877 1498 2014 10 4 1494 EWR
## 2: AA BOS 1173 1241 2014 4 15 1223 JFK
## 3: AA DFW 5877 1071 2014 6 13 1064 EWR
## 4: AA DFW 5877 1056 2014 9 12 1115 EWR
## 5: AA DFW 5877 1022 2014 6 16 1073 EWR
## ---
## 253312: WN BNA NA -12 2014 3 9 -21 LGA
## 253313: WN MDW NA -13 2014 3 10 -18 EWR
## 253314: WN HOU NA -13 2014 5 17 -30 LGA
## 253315: WN MKE NA -13 2014 6 15 10 LGA
## 253316: WN CAK NA -13 2014 8 19 -30 LGA
## air_time distance hour sum_delay carrierm_mean_arr
## 1: 200 1372 7 2992 5.541959
## 2: 39 NA 13 2464 1.903324
## 3: 175 1372 10 2135 8.690067
## 4: 198 1372 6 2171 -1.235160
## 5: 178 1372 7 2095 8.690067
## ---
## 253312: 115 764 16 -33 6.921642
## 253313: 112 711 20 -31 6.921642
## 253314: 202 1428 17 -43 22.875845
## 253315: 101 738 20 -3 14.888889
## 253316: 63 397 16 -43 7.219670
## carrierm_mean_dep
## 1: 7.591497
## 2: 3.987008
## 3: 11.476475
## 4: 3.307078
## 5: 11.476475
## ---
## 253312: 11.295709
## 253313: 11.295709
## 253314: 30.546453
## 253315: 24.217560
## 253316: 17.038047
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># inner join</span>
ans <- DT[carrierdest, <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># for each row in carrierdest</span>
nomatch = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0L</span>, <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># return only matching rows from both tables</span>
on = c(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"carrier"</span>,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dest"</span>)] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># joining on columns carrier and dest</span>
print(ans)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1071 2014 6 13 1064 EWR DFW 175
## 3: AA 1056 2014 9 12 1115 EWR DFW 198
## 4: AA 1022 2014 6 16 1073 EWR DFW 178
## 5: AA 989 2014 6 11 991 EWR DFW 194
## ---
## 23515: AA -8 2014 10 11 -13 JFK DCA 53
## 23516: AA -9 2014 5 21 -12 JFK DCA 52
## 23517: AA -9 2014 6 5 -6 JFK DCA 53
## 23518: AA -9 2014 10 2 -21 JFK DCA 51
## 23519: AA -11 2014 5 27 10 JFK DCA 55
## distance hour sum_delay carrierm_mean_arr carrierm_mean_dep count
## 1: 1372 7 2992 5.541959 7.591497 5877
## 2: 1372 10 2135 8.690067 11.476475 5877
## 3: 1372 6 2171 -1.235160 3.307078 5877
## 4: 1372 7 2095 8.690067 11.476475 5877
## 5: 1372 11 1980 8.690067 11.476475 5877
## ---
## 23515: NA 15 -21 5.541959 7.591497 172
## 23516: NA 15 -21 4.150172 8.733665 172
## 23517: NA 15 -15 8.690067 11.476475 172
## 23518: NA 15 -30 5.541959 7.591497 172
## 23519: NA 15 -1 4.150172 8.733665 172
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Be aware that because of the consistency to base R subsetting, the outer join is by default <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">RIGHT OUTER</code>. If we are looking for <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">LEFT OUTER</code>, we need to swap the tables, as in the example above. Exact behavior can also be easily controlled in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">merge</code> <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> method, using the same API as base R <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">merge</code> <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.frame</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we want to simply lookup the column(s) to our dataset, we can efficiently do it with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">:=</code> operator in <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">j</code>argument while joining. The same way as we sub-assign by reference, as described in the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Update dataset</em>section, we just now add a column by reference from the dataset to which we join. This avoids the in-memory copy of data, so we don’t need to assign results into new variables.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[carrierdest, <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># data.table to join with</span>
lkp.count := count, <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># lookup `count` column from `carrierdest`</span>
on = c(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"carrier"</span>,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dest"</span>)] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># join by columns</span>
head(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: AA 1498 2014 10 4 1494 EWR DFW 200
## 2: AA 1241 2014 4 15 1223 JFK BOS 39
## 3: AA 1071 2014 6 13 1064 EWR DFW 175
## 4: AA 1056 2014 9 12 1115 EWR DFW 198
## 5: AA 1022 2014 6 16 1073 EWR DFW 178
## 6: AA 989 2014 6 11 991 EWR DFW 194
## distance hour sum_delay carrierm_mean_arr carrierm_mean_dep lkp.count
## 1: 1372 7 2992 5.541959 7.591497 5877
## 2: NA 13 2464 1.903324 3.987008 1173
## 3: 1372 10 2135 8.690067 11.476475 5877
## 4: 1372 6 2171 -1.235160 3.307078 5877
## 5: 1372 7 2095 8.690067 11.476475 5877
## 6: 1372 11 1980 8.690067 11.476475 5877
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">aggregate while join</span>, use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">by = .EACHI</code>. It performs join that won’t materialize intermediate join results and will apply aggregates on the fly, making it memory efficient.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Rolling join</span> is an uncommon feature, designed for dealing with ordered data. It fits perfectly for processing temporal data, and time series in general. It basically roll matches in join condition to next matching value. Use it by providing the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">roll</code> argument when joining.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://github.com/Rdatatable/data.table/wiki/talks/EARL2014_OverlapRangeJoin_Arun.pdf" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Fast overlap join</span></a> joins datasets based on periods and its overlapping handling by using various overlaping operators: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">any</code>, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">within</code>, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">start</code>, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">end</code>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">non-equi join</span> feature to join datasets using non-equal condition is currently <a href="https://github.com/Rdatatable/data.table/issues/1452" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">being developed</a>.</div>
<h3 id="profiling-data" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Profiling Data</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When exploring our dataset, we may sometimes want to collect technical information on the subject, to better understand the quality of the data.</div>
<h4 id="descriptive-statistics" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Descriptive Statistics</h4>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">summary(DT)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month
## Length:253316 Min. :-112.00 Min. :2014 Min. : 1.000
## Class :character 1st Qu.: -5.00 1st Qu.:2014 1st Qu.: 3.000
## Mode :character Median : -1.00 Median :2014 Median : 6.000
## Mean : 12.47 Mean :2014 Mean : 5.639
## 3rd Qu.: 11.00 3rd Qu.:2014 3rd Qu.: 8.000
## Max. :1498.00 Max. :2014 Max. :10.000
##
## day arr_delay origin dest
## Min. : 1.00 Min. :-112.000 Length:253316 Length:253316
## 1st Qu.: 8.00 1st Qu.: -15.000 Class :character Class :character
## Median :16.00 Median : -4.000 Mode :character Mode :character
## Mean :15.89 Mean : 8.147
## 3rd Qu.:23.00 3rd Qu.: 15.000
## Max. :31.00 Max. :1494.000
##
## air_time distance hour sum_delay
## Min. : 20.0 Min. : 80.0 Min. : 0.00 Min. :-224.00
## 1st Qu.: 86.0 1st Qu.: 529.0 1st Qu.: 9.00 1st Qu.: -19.00
## Median :134.0 Median : 762.0 Median :13.00 Median : -5.00
## Mean :156.7 Mean : 950.4 Mean :13.06 Mean : 20.61
## 3rd Qu.:199.0 3rd Qu.:1096.0 3rd Qu.:17.00 3rd Qu.: 23.00
## Max. :706.0 Max. :4963.0 Max. :24.00 Max. :2992.00
## NA's :81483
## carrierm_mean_arr carrierm_mean_dep lkp.count
## Min. :-22.403 Min. :-4.500 Min. : 85
## 1st Qu.: 2.676 1st Qu.: 7.815 1st Qu.:3449
## Median : 6.404 Median :11.354 Median :5877
## Mean : 8.147 Mean :12.465 Mean :4654
## 3rd Qu.: 11.554 3rd Qu.:17.564 3rd Qu.:6058
## Max. : 86.182 Max. :52.864 Max. :6058
## NA's :229797
</code></pre>
<h4 id="cardinality" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Cardinality</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can check the <a href="https://en.wikipedia.org/wiki/Cardinality_%28SQL_statements%29" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">uniqueness of data</a> by using <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">uniqueN</code> function and apply it on every column. Object <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.SD</code> in the query below corresponds to <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">S</span>ubset of the <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">D</span>ata.table:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[, lapply(.SD, uniqueN)]
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: 14 570 1 10 31 616 3 109 509
## distance hour sum_delay carrierm_mean_arr carrierm_mean_dep lkp.count
## 1: 152 25 1021 134 134 11
</code></pre>
<h4 id="na-ratio" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
NA Ratio</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To calculate the ratio of unknown values (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">NA</code> in R, and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">NULL</code> in SQL) for each column, we provide the desired function to apply on every column.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">DT[, lapply(.SD, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span>(x) sum(is.na(x))/.N)]
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier dep_delay year month day arr_delay origin dest air_time
## 1: 0 0 0 0 0 0 0 0 0
## distance hour sum_delay carrierm_mean_arr carrierm_mean_dep lkp.count
## 1: 0.3216654 0 0 0 0 0.9071555
</code></pre>
<h3 id="exporting-data" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Exporting Data</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Fast export tabular data to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CSV</code> format</a> is also provided by the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> package.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-r hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">tmp.csv <- tempfile(fileext=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">".csv"</span>)
fwrite(DT, tmp.csv)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># preview exported data</span>
cat(system(paste(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"head -3"</span>,tmp.csv), intern=<span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">TRUE</span>), sep=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"\n"</span>)
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">## carrier,dep_delay,year,month,day,arr_delay,origin,dest,air_time,distance,hour,sum_delay,carrierm_mean_arr,carrierm_mean_dep,lkp.count
## AA,1498,2014,10,4,1494,EWR,DFW,200,1372,7,2992,5.54195933456561,7.59149722735674,5877
## AA,1241,2014,4,15,1223,JFK,BOS,39,,13,2464,1.90332441727168,3.98700802445548,1173
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
At the time of writing this, the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">fwrite</code> function hasn’t yet been published to the CRAN repository. To use it we need to <a href="https://github.com/Rdatatable/data.table/wiki/Installation" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">install <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> development version</a>, otherwise we can use base R <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">write.csv</code> function, but don’t expect it to be fast.</div>
<h2 id="resources" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Resources</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are plenty of resources available. Besides the manuals available for each function, there are also package vignettes, which are tutorials focused around the particular subject. Those can be found on the<a href="https://github.com/Rdatatable/data.table/wiki/Getting-started" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Getting started</a> page. Additionally, the <a href="https://github.com/Rdatatable/data.table/wiki/Presentations" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Presentations</a> page lists more than 30 materials (slides, video, etc.) from <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> presentations around the globe. Also, the community support has grown over the years, recently reaching the 4000-th question on Stack Overflow <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> tag, still having a high ratio (91.9%) of answered questions. The below plot presents the number of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> tagged questions on Stack Overflow over time.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="SO questions monthly for data.table - Only data.table tagged questions, not ones with data.table (accepted) answers" src="https://assets.toptal.io/uploads/blog/image/92638/toptal-blog-image-1465809558663-484733d8896538d7b3756369485b8fd1.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">SO questions monthly for data.table - Only data.table tagged questions, not ones with data.table (accepted) answers</span></div>
<h2 id="summary" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Summary</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article provides chosen examples for efficient tabular data transformation in R using the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code>package. The actual figures on performance can be examined by looking for reproducible benchmarks. I published a summarized blog post about <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> solutions for the top 50 rated StackOverflow questions for the R language called <a href="https://jangorecki.github.io/blog/2015-12-11/Solve-common-R-problems-efficiently-with-data.table.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Solve common R problems efficiently with data.table</a>, where you can find a lot of figures and reproducible code. The package <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">data.table</code> uses native implementation of fast radix ordering for its grouping operations, and binary search for fast subsets/joins. This radix ordering has been incorporated into base R from version <a href="https://stat.ethz.ch/pipermail/r-announce/2016/000602.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">3.3.0</a>. Additionally, the algorithm was recently implemented into H2O machine learning platform and parallelized over H2O cluster, <a href="http://library.fora.tv/2016/05/03/big_joins_scalable_data_munging_and_datatable" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">enabling efficient big joins on 10B x 10B rows</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a class="link is-blue" href="https://github.com/jangorecki" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3863a0; display: inline; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">JAN GORECKI</a>, a <a href="https://www.toptal.com/r/boost-your-data-munging-with-r" target="_blank">Toptal</a><a href="https://www.toptal.com/r" target="_blank"> freelance developer</a>.</div>
Anonymousnoreply@blogger.com2tag:blogger.com,1999:blog-8706416286757464743.post-71053061205728770272016-06-22T15:33:00.001+07:002016-06-22T15:33:04.410+07:00Web Audio API: Why Compose When You Can Code?<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The very first draft for a web audio API appeared in W3C during 2011. Although audio in webpages has been supported for a long time, a proper way of generating audio from the web browser was not available until quite recently. I personally attribute this to Google Chrome, because as for the interest of Google, the browser started becoming the most important part of a computer. You may recall, the realm of the web browser didn’t start to change much until Google Chrome appeared. If you used sound in a webpage in this time, it would have been a poor design decision. But since the idea of web experiments appeared, web audio started to make sense again. Web browsers nowadays are another tool for artistic expression, and video and audio in the web browser plays a vital role in it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Web Audio API: Why Compose When You Can Code?" src="https://assets.toptal.io/uploads/blog/image/92626/toptal-blog-image-1465474368121-ab12ddd46566e4737fd6587833e54ac9.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Web Audio API: Why Compose When You Can Code?</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Web Audio API can be quite hard to use for some purposes, as it is still under development, but a number of JavaScript libraries already exist to make things easier. In this case I am going to show you how to get started with the Web Audio API using a library called <a href="http://tonejs.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Tone.js</a>. With this, you will be able to cover most of your browser sound needs from only learning the basics.</div>
<h2 id="hello-web-audio-api" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
HELLO WEB AUDIO API</h2>
<h3 id="getting-started" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Getting Started</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We will begin without using the library. Our first experiment is going to involve making three sine waves. As this will be a simple example, we will create just one file named hello.html, a bare HTML file with a small amount of markup.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-html hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-doctype" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><!DOCTYPE html></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">html</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">head</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">meta</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">charset</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"utf‐8"</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">title</span>></span> Hello web audio <span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">title</span>></span> <span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">head</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">body</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">body</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span><span class="javascript" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">html</span>></span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The core of Web Audio API is the <a href="https://developer.mozilla.org/en/docs/Web/API/AudioContext" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">audio context</a>. The audio context is an object that will contain everything related to web audio. It’s not considered a good practice to have more than one audio context in a single project. We will begin by instantiating an audio context following the recommendations given by Mozilla’s Web Audio API documentation.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> audioCtx = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> (window.AudioContext || window.webkitAudioContext);
</code></pre>
<h3 id="making-an-oscillator" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Making an Oscillator</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With an audio context instantiated, you already have an audio component: the audioCtx.destination. This is like your speaker. To make a sound, you have to connect it to audioCtx.destination. Now to produce some sound, let’s create an oscillator:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sine = audioCtx.createOscillator();
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Great, but not enough. It also needs to be started and connected to our audioCtx.destination:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">sine.start();
sine.connect(audioCtx.destination);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With these four lines, you will have a pretty annoying webpage that plays a sine sound, but now you understand how modules can connect with one another. In the following script, there will be three sine shaped tone, connected to the output, each with a different tone. The code is very self-explanatory:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//create the context for the web audio</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> audioCtx = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> (window.AudioContext || window.webkitAudioContext)();
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//create, tune, start and connect each oscillator sinea, sineb and sinec</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sinea = audioCtx.createOscillator();
sinea.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">440</span>;
sinea.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sinea.start();
sinea.connect(audioCtx.destination);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sineb = audioCtx.createOscillator();
sineb.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">523.25</span>;
sineb.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sineb.start();
sineb.connect(audioCtx.destination);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sinec = audioCtx.createOscillator();
sinec.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">698.46</span>;
sinec.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sinec.start();
sinec.connect(audioCtx.destination);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Oscillators are not restricted to sine waves, but also can be triangles, sawtooth, square and custom shaped,<a href="https://developer.mozilla.org/en-US/docs/Web/API/OscillatorNode/type" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">as stated in the MDN</a>.</div>
<h3 id="patching-logic-of-web-audio" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Patching Logic of Web Audio</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Next, we will add a gain module to our orchestra of Web Audio components. This module allows us to change the amplitude of our sounds. It is akin to a volume knob. We have already used the connect function to connect an oscillator to the audio output. We can use the same connect function to connect any audio component. If you are using Firefox, and you take a look at the web audio console, you will see the following:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92624/toptal-blog-image-1465472992676-4f17407ee128886c8714bfe64ddd12b8.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we want to change the volume, our patch should look like:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92625/toptal-blog-image-1465473005483-a63bd2fe90b9fd09240952656b3e8f98.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Which means that the oscillators are no longer connected to the audio destination, but instead to a Gain module, and that gain module is connected to the destination. It’s good to always imagine that you do this with guitar pedals and cables. The code will look like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> audioCtx = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> (window.AudioContext || window.webkitAudioContext)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// we create the gain module, named as volume, and connect it to our</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> volume = audioCtx.createGain();
volume.connect(audioCtx.destination);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//these sines are the same, exept for the last connect statement.</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//Now they are connected to the volume gain module and not to the au</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sinea = audioCtx.createOscillator();
sinea.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">440</span>;
sinea.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sinea.start();
sinea.connect(volume);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sineb = audioCtx.createOscillator();
sineb.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">523.25</span>;
sineb.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sineb.start();
sineb.connect(volume);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sinec = audioCtx.createOscillator();
sinec.frequency.value = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">698.46</span>;
sinec.type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"sine"</span>;
sinec.start();
sinec.connect(volume);
volume.gain.value=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.2</span>;
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can find the solution at https://github.com/autotel/simple-webaudioapi/.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
GainNode is the most basic effect unit, but there is also a delay, a convolver, a biquadratic filter, a stereo panner, a wave shaper, and many others. You can grab new effects from libraries such as Tone.js.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Storing one of these sound patches in objects of their own will allow you to reuse them as needed, and create more complex orchestrations with less code. This could be a topic for a future post.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/developers/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/developers/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; line-height: 1.4; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-left-color: rgb(236, 236, 236); border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-style: solid none solid solid; border-top-color: rgb(236, 236, 236); border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><br /><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-left-color: rgb(236, 236, 236); border-style: solid none solid solid; border-top-color: rgb(236, 236, 236); border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-left-color: rgb(236, 236, 236); border-style: solid none solid solid; border-top-color: rgb(236, 236, 236); border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="making-things-easier-with-tonejs" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
MAKING THINGS EASIER WITH TONE.JS</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92627/toptal-blog-image-1465474393804-b2135b0069e586ba9b66690af9e65d8c.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we have taken a brief look how the vanilla Web Audio modules work, let us take a look at the awesome Web Audio framework: Tone.js. With this (and <a href="http://nexusosc.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">NexusUI</a> for user interface components), we can very easily build more interesting synths and sounds. To try things out, let us make a sampler and apply some user interactive effects to it, and then we will add some simple controls for this sample.</div>
<h3 id="tonejs-sampler" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tone.js Sampler</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can start by creating a simple project structure:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">simpleSampler
|-- js
|-- nexusUI.js
|-- Tone.js
|-- noisecollector_hit4.wav
|-- sampler.html
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Our JavaScript libraries will reside in the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">js</em> directory. For the purposes of this demo, we can use<a href="https://freesound.org/people/NoiseCollector/sounds/113287/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">NoiseCollector’s hit4.wav</a> file that can be downloaded from Freesound.org.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Tone.js provides its functionalities through Player objects. The basic capability of the object is to load a sample, and to play it either in a loop or once. Our first step here is to create a player object in a “sampler” var, inside the sampler.html file:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-html hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-doctype" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><!doctype html></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">html</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">head</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">title</span>></span> Sampler <span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">title</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">type</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"text/javascript"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">src</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"js/nexusUI.js"</span> ></span><span class="javascript" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">type</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"text/javascript"</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">src</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"js/Tone.js"</span> ></span><span class="javascript" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span><span class="javascript" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> sampler = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Tone.Player(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"noisecollector_hit4.wav"</span>, <span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
console.log(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"samples loaded"</span>);
});
</span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">script</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">head</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">body</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">body</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">html</span>></span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that the first parameter of the player constructor is the name of the <a href="https://en.wikipedia.org/wiki/WAV" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">WAV file</a>, and the second is a callback function. WAV is not the only supported file type, and the compatibility depends on the web browser more than the library. The callback function will run when the player has finished loading the sample into its buffer.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We also have to connect our sampler to the output. The Tone.js way of doing this is:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">sampler.toMaster();
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
… where sampler is a Tone.Player object, after line 10. The toMaster function is shorthand for connect(Tone.Master).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you open your web browser with the developer console open, you should see the “samples loaded” message, indicating that the player was created correctly. At this point you may want to hear the sample. To do that, we need to add a button to the webpage, and program it to play the sample once pressed. We are going to use a NexusUI button in the body:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-html hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">nx</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"button"</span>></span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span>></span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should now see a rounded button being rendered in the document. To program it to play our sample, we add a NexusUI listener, which looks like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">button1.on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'*'</span>,<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
console.log(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"button pressed!"</span>);
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Something outstanding about NexusUI is that it creates a global variable for each NexusUI element. You can set NexusUI to not do that, and instead have these variables only in nx.widgets[] by setting nx.globalWidgets to false. Here we are going to create just a couple of elements, so we’ll just stick to this behaviour.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Same as in jQuery, we can put these .on events, and the first argument will be the event name. Here we are just assigning a function to whatever is done to the button. This whatever is written as “*”. You can learn more about events for each element in the NexusUI API. To play the sample instead of logging messages when we press the button, we should run the start function of our sampler.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">nx.onload = <span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
button1.on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'*'</span>,<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
console.log(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"button pressed!"</span>);
sampler.start();
});
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Also notice that the listener goes inside an onload callback. NexusUI elements are drawn in canvas, and you can’t refer to them until nx calls the onload function. Just as you would do with DOM elements in jQuery.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The event is triggered on mouse down and on release. If you want it to be triggered only on press, you have to evaluate whether event.press equals one.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With this, you should have a button that plays the sample on each press. If you set sampler.retrigger to true, it will allow you to play the sample regardless of whether it is playing or not. Otherwise, you have to wait until the sample finishes to retrigger it.</div>
<h3 id="applying-effects" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Applying Effects</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With Tone.js, we can easily create a delay:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">var</span> delay= <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Tone.FeedbackDelay(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"16n"</span>,<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.5</span>).toMaster();
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first argument is the delay time, which can be written in musical notation as shown here. The second is the wet level, which means the mixture between the original sound and the sound that has an effect on it. For delays you don’t usually want a 100% wet, because delays are interesting with respect to the original sound, and the wet alone is not very appealing as both together.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The next step is to unplug our sampler from master and plug it instead to the delay. Tweak the line where sampler is connected to master:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">sampler.connect(delay);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now try the button again and see the difference.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Next, we will add two dials to the body of our document:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-html hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">nx</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dial"</span>></span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span>></span>
<span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">nx</span>=<span class="hljs-value" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"dial"</span>></span><span class="hljs-tag" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"></<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">canvas</span>></span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And we apply the dials’ values to the delay effect using the NexusUIlistener:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">dial1.on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'*'</span>,<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
delay.delayTime.value=data.value;
})
dial2.on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'*'</span>,<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
delay.feedback.value=data.value;
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The parameters that you can tweak on each event can be found in Tone.js documentations. For delay, it is<a href="http://tonejs.org/docs/#FeedbackDelay" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">here</a>. Now you are ready to try the example and tweak the delay parameters with the NexusUI dials. This process can be easily done with each NexusUI element, not limited only to effects. For instance, also try adding another dial, and adding its listener as follows:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">dial3.on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'*'</span>,<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
sampler.playbackRate=data.value;
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can find these files at <a href="https://github.com/autotel/simpleSampler/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">github.com/autotel/simpleSampler</a></div>
<h2 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
CONCLUSION</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I went through these APIs, I started feeling overwhelmed by all the possibilities and ideas that started to come to my mind. The big difference between this implementation of audio and the traditional implementations of digital audio is not in the audio itself, but in the context. There are no new methods of making synthesis here. Rather the innovation is that audio and music making are now meeting <a href="https://www.toptal.com/web" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">web technologies</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I am personally involved in electronic music making, and this area has always had this paradox of the ambiguity between actually performing music and just pressing play to a recorded track. If you want to really make live electronic music, you must be able to create your own performative tools or “music-making robots” for live improvisation. But if the performance of electronic music becomes simply tweaking parameters in pre-prepared music making algorithms, then the audience can also be involved in this process. I have been working on little experiments regarding this integration of web and audio for crowdsourced music, and perhaps soon we will be attending parties where the music comes from the audience through their smartphones. After all, it’s not that different from rhythmic jams we might have enjoyed in the cave ages.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a class="link is-blue" href="https://www.toptal.com/resume/joaquin-aldunate" style="border: 0px; box-sizing: border-box; color: #3863a0; display: inline; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">JOAQUÍN ALDUNATE</a>, <span style="font-size: 19.2px; line-height: 28.8px;">a <a href="https://www.toptal.com/web/web-audio-api-tutorial" style="color: blue; text-decoration: none;" target="_blank">Toptal</a><a href="https://www.toptal.com/html5" style="color: blue; text-decoration: none;" target="_blank"> HTML </a></span></div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-57139329295127719822016-06-14T17:07:00.000+07:002016-06-14T17:07:15.674+07:00Write Code to Rewrite Your Code: jscodeshift<h2 id="codemods-with-jscodeshift" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Codemods with jscodeshift</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How many times have you used the find-and-replace functionality across a directory to make changes to JavaScript source files? If you’re good, you’ve gotten fancy and used regular expressions with capturing groups, because it’s worth the effort if your code base is sizable. Regex has limits, though. For non-trivial changes you need a developer who understands the code in context and is also willing to take on the long, tedious, and error-prone process.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is where “codemods” come in.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Codemods are scripts used to rewrite other scripts. Think of them as a find and replace functionality that can read and write code. You can use them to update source code to fit a team’s coding conventions, make widespread changes when an API is modified, or even auto-fix existing code when your public package makes a breaking change.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="The jscodeshift toolkit is great for working with codemods." src="https://assets.toptal.io/uploads/blog/image/92595/toptal-blog-image-1464964321406-0ab31a1320d562c3f58fab28be8eb8e8.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Think of codemods as a scripted find and replace functionality that can read and write code.</span></div>
<div class="tweet_this" style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: -10px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this article, we’re going to explore a toolkit for codemods called “jscodeshift” while creating three codemods of increasing complexity. By the end you will have broad exposure to the important aspects of<a href="https://github.com/facebook/jscodeshift/wiki/jscodeshift-Documentation" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">jscodeshift</a> and will be ready to start writing your own codemods. We will go through three exercises that cover some basic, but awesome, uses of codemods, and you can view the source code for these exercises on my <a href="https://github.com/reergymerej/jscodeshift-article" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">github project</a>.</div>
<h2 id="what-is-jscodeshift" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What Is jscodeshift?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The jscodeshift toolkit allows you to pump a bunch of source files through a transform and replace them with what comes out the other end. Inside the transform, you parse the source into an abstract syntax tree (AST), poke around to make your changes, then regenerate the source from the altered AST.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The interface that jscodeshift provides is a wrapper around <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">recast</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ast-types</code> packages. <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">recast</code>handles the conversion from source to AST and back while <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ast-types</code> handles the low-level interaction with the AST nodes.</div>
<h3 id="setup" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Setup</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get started, install jscodeshift globally from npm.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-sh" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">npm i -g jscodeshift
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are runner options you can use and an opinionated test setup that makes running a suite of tests via Jest (an open source JavaScript testing framework) really easy, but we’re going to bypass that for now in favor of simplicity:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t some-transform.js input-file.js -d -p</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This will run <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">input-file.js</code> through the transform <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">some-transform.js</code> and print the results without altering the file.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Before jumping in, though, it is important to understand three main object types that the jscodeshift API deals with: nodes, node-paths, and collections.</div>
<h3 id="nodes" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Nodes</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Nodes are the basic building blocks of the AST, often referred to as “AST nodes.” These are what you see when exploring your code with AST Explorer. They are simple objects and do not provide any methods.</div>
<h3 id="node-paths" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Node-paths</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Node-paths are wrappers around an AST node provided by <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ast-types</code> as a way to traverse the abstract syntax tree (AST, remember?). In isolation, nodes do not have any information about their parent or scope, so node-paths take care of that. You can access the wrapped node via the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">node</code> property and there are several methods available to change the underlying node. node-paths are often referred to as just “paths.”</div>
<h3 id="collections" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Collections</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Collections are groups of zero or more node-paths that the jscodeshift API returns when you query the AST. They have all sorts of useful methods, some of which we will explore.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Collections contain node-paths, node-paths contain nodes, and nodes are what the AST is made of. Keep that in mind and it will be easy to understand the jscodeshift query API.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It can be tough to keep track of the differences between these objects and their respective API capabilities, so there’s a nifty tool called <a href="https://www.npmjs.com/package/jscodeshift-helper" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">jscodeshift-helper</a> that logs the object type and provides other key information.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Knowing the difference between nodes, node-paths, and collections is important." src="https://assets.toptal.io/uploads/blog/image/92596/toptal-blog-image-1464964398499-230b260539e4ac343918dc08208303c7.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Knowing the difference between nodes, node-paths, and collections is important.</span></div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://draft.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://draft.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://draft.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="exercise-1---remove-calls-to-console" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Exercise 1 - Remove Calls To console</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get our feet wet, let’s start with removing calls to all console methods in our code base. While you can do this with find and replace and a little regex, it starts to get tricky with multiline statements, template literals, and more complex calls, so it’s an ideal example to start with.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
First, create two files, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">remove-consoles.js</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">remove-consoles.input.js</code>:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//remove-consoles.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
};
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//remove-consoles.input.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> sum = (a, b) => {
console.log(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'calling sum with'</span>, <span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">arguments</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a + b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> multiply = (a, b) => {
console.warn(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'calling multiply with'</span>,
<span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">arguments</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a * b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> divide = (a, b) => {
console.error(`calling divide <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">with</span> ${ <span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">arguments</span> }`);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a / b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> average = (a, b) => {
console.log(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'calling average with '</span> + <span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">arguments</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> divide(sum(a, b), <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>);
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here’s the command we’ll be using in the terminal to push it through jscodeshift:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t remove-consoles.js remove-consoles.input.js -d -p</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If everything’s set up correctly, when you run it you should see something like this.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Processing 1 files...
Spawning 1 workers...
Running in dry mode, no files will be written!
Sending 1 files to free worker...
All done.
Results:
0 errors
0 unmodified
1 skipped
0 ok
Time elapsed: 0.514seconds
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
OK, that was a bit anticlimactic since our transform doesn’t actually do anything yet, but at least we know it’s all working. If it doesn’t run at all, make sure you installed jscodeshift globally. If the command to run the transform is incorrect, you’ll either see an “ERROR Transform file … does not exist” message or “TypeError: path must be a string or Buffer” if the input file cannot be found. If you’ve fat-fingered something, it should be easy to spot with the very descriptive transformation errors.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: 1.2em; line-height: 1.5em; text-align: center;">Our end goal though, after a successful transform, is to see this source:</span></div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> sum = (a, b) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a + b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> multiply = (a, b) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a * b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> divide = (a, b) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> a / b;
};
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> average = (a, b) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> divide(sum(a, b), <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>);
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To get there, we need to convert the source into an AST, find the consoles, remove them, and then convert the altered AST back into source. The first and last steps are easy, it’s just:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">remove-consoles.js
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But how do we find the consoles and remove them? Unless you have some exceptional knowledge of the Mozilla Parser API, you’ll probably need a tool to help understand what the AST looks like. For that you can use the <a href="http://astexplorer.net/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">AST Explorer</a>. Paste the contents of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">remove-consoles.input.js</code> into it and you’ll see the AST. There is a lot of data even in the simplest code, so it helps to hide location data and methods. You can toggle the visibility of properties in AST Explorer with the checkboxes above the tree.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can see that calls to console methods are referred to as <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CallExpressions</code>, so how do we find them in our transform? We use jscodeshift’s queries, remembering our earlier discussion on the differences between Collections, node-paths and nodes themselves:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//remove-consoles.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The line <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">const root = j(fileInfo.source);</code> returns a collection of one node-path, which wraps the root AST node. We can use the collection’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">find</code> method to search for descendant nodes of a certain type, like so:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> callExpressions = root.find(j.CallExpression);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This returns another collection of node-paths containing just the nodes that are CallExpressions. At first blush, this seems like what we want, but it is too broad. We might end up running hundreds or thousands of files through our transforms, so we have to be precise to have any confidence that it will work as intended. The naive <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">find</code> above would not just find the console CallExpressions, it would find every CallExpression in the source, including</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">require</span>(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'foo'</span>)
bar()
setTimeout(() => {}, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>)
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To force greater specificity, we provide a second argument to <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.find</code>: An object of additional parameters, each node needs to be included in the results. We can look at the AST Explorer to see that our console.* calls have the form of:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">{
"type": "CallExpression",
"callee": {
"type": "MemberExpression",
"object": {
"type": "Identifier",
"name": "console"
}
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With that knowledge, we know to refine our query with a specifier that will return only the type of CallExpressions we’re interested in:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">const callExpressions = root.find(j.CallExpression, {
callee: {
type: 'MemberExpression',
object: { type: 'Identifier', name: 'console' },
},
});
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we’ve got an accurate collection of the call sites, let’s remove them from the AST. Conveniently, the collection object type has a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">remove</code> method that will do just that. Our <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">remove-consoles.js</code> file will now look like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//remove-consoles.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> callExpressions = root.find(j.CallExpression, {
callee: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'MemberExpression'</span>,
object: { type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Identifier'</span>, name: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'console'</span> },
},
}
);
callExpressions.remove();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, if we run our transform from the command line using <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t remove-consoles.js remove-consoles.input.js -d -p</code>, we should see:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Processing 1 files...
Spawning 1 workers...
Running in dry mode, no files will be written!
Sending 1 files to free worker...
export const sum = (a, b) => {
return a + b;
};
export const multiply = (a, b) => {
return a * b;
};
export const divide = (a, b) => {
return a / b;
};
export const average = (a, b) => {
return divide(sum(a, b), 2);
};
All done.
Results:
0 errors
0 unmodified
0 skipped
1 ok
Time elapsed: 0.604seconds
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It looks good. Now that our transform alters the underlying AST, using <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.toSource()</code> generates a different string from the original. The -p option from our command displays the result, and a tally of dispositions for each file processed is shown at the bottom. Removing the -d option from our command, would replace the content of remove-consoles.input.js with the output from the transform.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Our first exercise is complete… almost. The code is bizarre looking and probably very offensive to any functional purists out there, and so to make transform code flow better, jscodeshift has made most things chainable. This allows us to rewrite our transform like so:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// remove-consoles.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> j(fileInfo.source)
.find(j.CallExpression, {
callee: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'MemberExpression'</span>,
object: { type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Identifier'</span>, name: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'console'</span> },
},
}
)
.remove()
.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Much better. To recap exercise 1, we wrapped the source, queried for a collection of node-paths, change the AST, and then regenerated that source. We’ve gotten our feet wet with a pretty simple example and touched on the most important aspects. Now, let’s do something more interesting.</div>
<h2 id="exercise-2---replacing-imported-method-calls" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Exercise 2 - Replacing Imported Method Calls</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For this scenario, we’ve got a “geometry” module with a method named “circleArea” that we’ve deprecated in favor of “getCircleArea.” We could easily find and replace these with <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">/geometry\.circleArea/g</code>, but what if the user has imported the module and assigned it a different name? For example:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">import g from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'geometry'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> area = g.circleArea(radius);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How would we know to replace <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">g.circleArea</code> instead of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">geometry.circleArea</code>? We certainly cannot assume that all <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">circleArea</code> calls are the ones we’re looking for, we need some context. This is where codemods start showing their value. Let’s start by making two files, <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">deprecated.js</code> and <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">deprecated.input.js</code>.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//deprecated.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.toSource();
};
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">deprecated.input.js
import g from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'geometry'</span>;
import otherModule from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'otherModule'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> radius = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">20</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> area = g.circleArea(radius);
console.log(area === <span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Math</span>.pow(g.getPi(), <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>) * radius);
console.log(area === otherModule.circleArea(radius));
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now run this command to run the codemod.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t ./deprecated.js ./deprecated.input.js -d -p</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You should see output indicating the transform ran, but has not changed anything yet.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-sh" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Processing 1 files...
Spawning 1 workers...
Running in dry mode, no files will be written!
Sending 1 files to free worker...
All done.
Results:
0 errors
1 unmodified
0 skipped
0 ok
Time elapsed: 0.892seconds
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We need to know what our <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">geometry</code> module has been imported as. Let’s look at the AST Explorer and figure out what we’re looking for. Our import takes this form.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">{
"type": "ImportDeclaration",
"specifiers": [
{
"type": "ImportDefaultSpecifier",
"local": {
"type": "Identifier",
"name": "g"
}
}
],
"source": {
"type": "Literal",
"value": "geometry"
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can specify an object type to find a collection of nodes like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> importDeclaration = root.find(j.ImportDeclaration, {
source: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Literal'</span>,
value: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'geometry'</span>,
},
});
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This gets us the ImportDeclaration used to import “geometry”. From there, dig down to find the local name used to hold the imported module. Since this is the first time we’ve done it, let’s point out an important and confusing point when first starting.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note: It’s important to know that <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">root.find()</code> returns a collection of node-paths. From there, the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.get(n)</code>method returns the node-path at index <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">n</code> in that collection, and to get the actual node, we use <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.node</code>. The node is basically what we see in AST Explorer. Remember, the node-path is mostly information about the scope and relationships of the node, not the node itself.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find the Identifiers</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> identifierCollection = importDeclaration.find(j.Identifier);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the first NodePath from the Collection</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> nodePath = identifierCollection.get(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the Node in the NodePath and grab its "name"</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> localName = nodePath.node.name;
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This allows us to figure out dynamically what our <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">geometry</code> module has been imported as. Next, we find the places it is being used and change them. By looking at AST Explorer, we can see that we need to find MemberExpressions that look like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">{
"type": "MemberExpression",
"object": {
"name": "geometry"
},
"property": {
"name": "circleArea"
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Remember, though, that our module may have been imported with a different name, so we have to account for that by making our query look like this instead:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">j.MemberExpression, {
object: {
name: localName,
},
property: {
name: "circleArea",
},
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we have a query, we can get a collection of all the call sites to our old method and then use the collection’s <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">replaceWith()</code> method to swap them out. The <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">replaceWith()</code> method iterates through the collection, passing each node-path to a callback function. The AST Node is then replaced with whatever Node you return from the callback.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Codemods allow you to script 'intelligent' considerations for refactoring." src="https://assets.toptal.io/uploads/blog/image/92597/toptal-blog-image-1464964465005-21dc716fb95a5de71eceeb76137358ed.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Again, understanding the difference between collections, node-paths and nodes is necessary for this to make sense.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once we’re done with the replacement, we generate the source as usual. Here’s our finished transform:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//deprecated.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find declaration for "geometry" import</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> importDeclaration = root.find(j.ImportDeclaration, {
source: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Literal'</span>,
value: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'geometry'</span>,
},
});
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the local name for the imported module</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> localName =
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find the Identifiers</span>
importDeclaration.find(j.Identifier)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the first NodePath from the Collection</span>
.get(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the Node in the NodePath and grab its "name"</span>
.node.name;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.find(j.MemberExpression, {
object: {
name: localName,
},
property: {
name: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'circleArea'</span>,
},
})
.replaceWith(nodePath => {
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the underlying Node</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> { node } = nodePath;
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// change to our new prop</span>
node.property.name = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'getCircleArea'</span>;
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// replaceWith should return a Node, not a NodePath</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> node;
})
.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When we run the source through the transform, we see that the call to the deprecated method in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">geometry</code> module was changed, but the rest was left unaltered, like so:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">import g from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'geometry'</span>;
import otherModule from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'otherModule'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> radius = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">20</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> area = g.getCircleArea(radius);
console.log(area === <span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Math</span>.pow(g.getPi(), <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>) * radius);
console.log(area === otherModule.circleArea(radius));
</code></pre>
<h2 id="exercise-3---changing-a-method-signature" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Exercise 3 - Changing A Method Signature</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the previous exercises we covered querying collections for specific types of nodes, removing nodes, and altering nodes, but what about creating altogether new nodes? That’s what we’ll tackle in this exercise.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this scenario, we’ve got a method signature that’s gotten out of control with individual arguments as the software has grown, and so it has been decided it would be better to accept an object containing those arguments instead.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Instead of <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">car.factory('white', 'Kia', 'Sorento', 2010, 50000, null, true);</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
we’d like to see</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> suv = car.factory({
color: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'white'</span>,
make: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Kia'</span>,
model: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Sorento'</span>,
year: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2010</span>,
miles: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">50000</span>,
bedliner: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>,
alarm: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>,
});
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s start by making the transform and an input file to test with:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//signature-change.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.toSource();
};
</code></pre>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//signature-change.input.js</span>
import car from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'car'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> suv = car.factory(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'white'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Kia'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Sorento'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2010</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">50000</span>, <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>, <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> truck = car.factory(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'silver'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Toyota'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Tacoma'</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2006</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>, <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>, <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Our command to run the transform will be <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t signature-change.js signature-change.input.js -d -p</code> and the steps we need to perform this transformation are:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Find the local name for the imported module</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Find all call sites to the .factory method</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Read all arguments being passed in</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Replace that call with a single argument which contains an object with the original values</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Using the AST Explorer and the process we used in the previous exercises, the first two steps are easy:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//signature-change.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find declaration for "car" import</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> importDeclaration = root.find(j.ImportDeclaration, {
source: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Literal'</span>,
value: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'car'</span>,
},
});
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the local name for the imported module</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> localName =
importDeclaration.find(j.Identifier)
.get(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>)
.node.name;
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find where `.factory` is being called</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.find(j.CallExpression, {
callee: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'MemberExpression'</span>,
object: {
name: localName,
},
property: {
name: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'factory'</span>,
},
}
})
.toSource();
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For reading all the arguments currently being passed in, we use the<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">replaceWith()</code> method on our collection of CallExpressions to swap each of the nodes. The new nodes will replace node.arguments with a new single argument, an object.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Easily swap method arguments with jscodeshift!" src="https://assets.toptal.io/uploads/blog/image/92598/toptal-blog-image-1464964526266-0a6a71f0386daf6dc8d9df4d99cd4efd.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Change method signatures with 'replacewith()' and swap out entire nodes.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s give it a try with a simple object to make sure we know how this works before we use the proper values:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> .replaceWith(nodePath => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> { node } = nodePath;
node.arguments = [{ foo: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'bar'</span> }];
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> node;
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When we run this (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t signature-change.js signature-change.input.js -d -p</code>), the transform will blow up with:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> ERR signature-change.input.js Transformation error
Error: {foo: bar} does not match type Printable
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Turns out we can’t just jam plain objects into our AST nodes. Instead, we need to use builders to create proper nodes.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #3863a0; font-size: 1.3em; line-height: 1.3em; text-align: center;">Node Builders</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Builders allow us to create new nodes properly; they are provided by <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ast-types</code> and surfaced through jscodeshift. They rigidly check that the different types of nodes are created correctly, which can be frustrating when you’re hacking away on a roll, but ultimately, this is a good thing. To understand how to use builders, there are two things you should keep in mind:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All of the available AST node types are defined in the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">def</code>folder of the <a href="https://github.com/benjamn/ast-types/tree/master/def" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ast-types github project</a>, mostly in core.js There are builders for all the AST node types, but they use <a href="https://msdn.microsoft.com/en-us/library/x2dbyw72(v=vs.71).aspx" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">camel-cased</a> version of the node type, not<a href="http://c2.com/cgi/wiki?PascalCase" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pascal-case</a>. (This isn’t explicitly stated, but you can see this is the case in the <a href="https://github.com/benjamn/ast-types/blob/master/lib/types.js#L609" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ast-types source</a></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we use AST Explorer with an example of what we want the result to be, we can piece this together pretty easily. In our case, we want the new single argument to be an ObjectExpression with a bunch of properties. Looking at the type definitions mentioned above, we can see what this entails:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"ObjectExpression"</span>)
.bases(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Expression"</span>)
.build(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"properties"</span>)
.field(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"properties"</span>, [def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Property"</span>)]);
def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Property"</span>)
.bases(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Node"</span>)
.build(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"kind"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"key"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"value"</span>)
.field(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"kind"</span>, or(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"init"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"get"</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"set"</span>))
.field(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"key"</span>, or(def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Literal"</span>), def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Identifier"</span>)))
.field(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"value"</span>, def(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Expression"</span>));
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, the code to build an AST node for { foo: ‘bar’ } would look like:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">j.objectExpression([
j.property(
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'init'</span>,
j.identifier(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'foo'</span>),
j.literal(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'bar'</span>)
)
]);
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Take that code and plug it into our transform like so:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">.replaceWith(nodePath => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> { node } = nodePath;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> object = j.objectExpression([
j.property(
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'init'</span>,
j.identifier(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'foo'</span>),
j.literal(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'bar'</span>)
)
]);
node.arguments = [object];
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> node;
})
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Running this gets us the result:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">import car from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'car'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> suv = car.factory({
foo: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"bar"</span>
});
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> truck = car.factory({
foo: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"bar"</span>
});
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that we know how to create a proper AST node, it’s easy to loop through the old arguments and generate a new object to use, instead. Here’s what our <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">signature-change.js</code> file looks like now:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//signature-change.js</span>
export <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">default</span> (fileInfo, api) => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> j = api.jscodeshift;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> root = j(fileInfo.source);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find declaration for "car" import</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> importDeclaration = root.find(j.ImportDeclaration, {
source: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Literal'</span>,
value: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'car'</span>,
},
});
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// get the local name for the imported module</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> localName =
importDeclaration.find(j.Identifier)
.get(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>)
.node.name;
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// current order of arguments</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> argKeys = [
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'color'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'make'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'model'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'year'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'miles'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'bedliner'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'alarm'</span>,
];
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// find where `.factory` is being called</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> root.find(j.CallExpression, {
callee: {
type: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'MemberExpression'</span>,
object: {
name: localName,
},
property: {
name: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'factory'</span>,
},
}
})
.replaceWith(nodePath => {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> { node } = nodePath;
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// use a builder to create the ObjectExpression</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> argumentsAsObject = j.objectExpression(
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// map the arguments to an Array of Property Nodes</span>
node.arguments.map((arg, i) =>
j.property(
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'init'</span>,
j.identifier(argKeys[i]),
j.literal(arg.value)
)
)
);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// replace the arguments with our new ObjectExpression</span>
node.arguments = [argumentsAsObject];
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> node;
})
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// specify print options for recast</span>
.toSource({ quote: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'single'</span>, trailingComma: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span> });
};
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Run the transform (<code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">jscodeshift -t signature-change.js signature-change.input.js -d -p</code>) and we’ll see the signatures have been updated as expected:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">import car from <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'car'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> suv = car.factory({
color: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'white'</span>,
make: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Kia'</span>,
model: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Sorento'</span>,
year: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2010</span>,
miles: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">50000</span>,
bedliner: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">null</span>,
alarm: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>,
});
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> truck = car.factory({
color: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'silver'</span>,
make: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Toyota'</span>,
model: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Tacoma'</span>,
year: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2006</span>,
miles: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>,
bedliner: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>,
alarm: <span class="hljs-literal" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>,
});
</code></pre>
<h2 id="codemods-with-jscodeshift-recap" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: "Proxima Nova", Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Codemods With jscodeshift Recap</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It took a little time and effort to get to this point, but the benefits are huge when faced with mass refactoring. Distributing groups of files to different processes and running them in parallel is something jscodeshift excels at, allowing you to run complex transformations across a huge codebase in seconds. As you become more proficient with codemods, you’ll start repurposing existing scripts (such as the <a href="https://github.com/reactjs/react-codemod" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">react-codemod github repository</a> or writing your own for all sorts of tasks, and that will make you, your team, and your package-users more efficient.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: "Proxima Nova", Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a href="https://www.toptal.com/resume/jeremy-greer" style="background-color: #436ba8; border: 0px; box-sizing: border-box; color: white; font-size: 19px; font-weight: 600; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; text-decoration: none; vertical-align: baseline;">Jeremy Greer</a>, a <a href="https://www.toptal.com/javascript" target="_blank">Toptal</a><a href="https://www.toptal.com/javascript/write-code-to-rewrite-your-code" target="_blank"> JavaScript developer</a>.</div>
Anonymousnoreply@blogger.com2tag:blogger.com,1999:blog-8706416286757464743.post-27734245503715763412016-06-08T19:36:00.002+07:002016-06-08T19:36:42.525+07:00Clustering Algorithms: From Start To State Of The Art<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It’s not a bad time to be a Data Scientist. Serious people may find interest in you if you turn the conversation towards “Big Data”, and the rest of the party crowd will be intrigued when you mention “Artificial Intelligence” and “Machine Learning”. Even <a href="https://www.google.com/trends/explore#q=%2Fm%2F0c_xl%2C%20Data%20Scientist%2C%20Data%20Miner&cmpt=q&tz=Etc%2FGMT-6" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Google thinks</a> you’re not bad, and that you’re getting even better. There are a lot of ‘smart’ algorithms that help data scientists do their wizardry. It may all seem complicated, but if we understand and organize algorithms a bit, it’s not even that hard to find and apply the one that we need.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92521/toptal-blog-image-1463639098844-eb9ad14c7f665e556b2cb66a65b6c257.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Courses on data mining or machine learning will usually start with clustering, because it is both simple and useful. It is an important part of a somewhat wider area of Unsupervised Learning, where the data we want to describe is not labeled. In most cases this is where the user did not give us much information of what is the expected output. The algorithm only has the data and it should do the best it can. In our case, it should perform clustering – separating data into groups (clusters) that contain similar data points, while the dissimilarity between groups is as high as possible. Data points can represent anything, such as our clients. Clustering can be useful if we, for example, want to group similar users and then run different marketing campaigns on each cluster.</div>
<h2 id="k-means-clustering" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
K-Means Clustering</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
After the necessary introduction, Data Mining courses always continue with K-Means; an effective, widely used, all-around clustering algorithm. Before actually running it, we have to define a distance function between data points (for example, Euclidean distance if we want to cluster points in space), and we have to set the number of clusters we want (k).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92528/toptal-blog-image-1463672901961-c86610183bb2ba67f979c421f6748893.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The algorithm begins by selecting k points as starting centroids (‘centers’ of clusters). We can just select any k random points, or we can use some other approach, but picking random points is a good start. Then, we iteratively repeat two steps:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Assignment step: each of m points from our dataset is assigned to a cluster that is represented by the closest of the k centroids. For each point, we calculate distances to each centroid, and simply pick the least distant one.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Update step: for each cluster, a new centroid is calculated as the mean of all points in the cluster. From the previous step, we have a set of points which are assigned to a cluster. Now, for each such set, we calculate a mean that we declare a new centroid of the cluster.</div>
</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
After each iteration, the centroids are slowly moving, and the total distance from each point to its assigned centroid gets lower and lower. The two steps are alternated <a href="http://simplystatistics.org/2014/02/18/k-means-clustering-in-a-gif/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">until convergence</a>, meaning until there are no more changes in cluster assignment. After a number of iterations, the same set of points will be assigned to each centroid, therefore leading to the same centroids again. K-Means is guaranteed to converge to a local optimum. However, that does not necessarily have to be the best overall solution (global optimum).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The final clustering result can depend on the selection of initial centroids, so a lot of thought has been given to this problem. One simple solution is just to run K-Means a couple of times with random initial assignments. We can then select the best result by taking the one with the minimal sum of distances from each point to its cluster – the error value that we are trying to minimize in the first place.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Other approaches to selecting initial points can rely on selecting distant points. This can lead to better results, but we may have a problem with outliers, those rare alone points that are just “off” that may just be some errors. Since they are far from any meaningful cluster, each such point may end up being its own ‘cluster’. A good balance is <a href="https://en.wikipedia.org/wiki/K-means%2B%2B" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">K-Means++</a> variant [Arthur and Vassilvitskii, 2007], whose initialization will still pick random points, but with probability proportional to square distance from the previously assigned centroids. Points that are further away will have higher probability to be selected as starting centroids. Consequently, if there’s a group of points, the probability that a point from the group will be selected also gets higher as their probabilities add up, resolving the outlier problem we mentioned.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
K-Means++ is also the default initialization for Python’s <a href="http://scikit-learn.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Scikit-learn</a> K-Means implementation. If you’re using Python, this may be your library of choice. For Java, <a href="http://www.cs.waikato.ac.nz/ml/weka/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Weka</a> library may be a good start:</div>
<h4 id="java-weka" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Java (Weka)</h4>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Load some data</span>
Instances data = DataSource.read(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"data.arff"</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Create the model</span>
SimpleKMeans kMeans = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> SimpleKMeans();
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// We want three clusters</span>
kMeans.setNumClusters(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Run K-Means</span>
kMeans.buildClusterer(data);
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Print the centroids</span>
Instances centroids = kMeans.getClusterCentroids();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (Instance centroid: centroids) {
System.out.println(centroid);
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Print cluster membership for each instance</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (Instance point: data) {
System.out.println(point + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">" is in cluster "</span> + kMeans.clusterInstance(point));
}
</code></pre>
<h4 id="python-scikit-learn" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Python (Scikit-learn)</h4>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-py" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">>>> from sklearn import cluster, datasets
>>> iris = datasets.load_iris()
>>> X_iris = iris.data
>>> y_iris = iris.target
>>> k_means = cluster.KMeans(n_clusters=3)
>>> k_means.fit(X_iris)
KMeans(copy_x=True, init='k-means++', ...
>>> print(k_means.labels_[::10])
[1 1 1 1 1 0 0 0 0 0 2 2 2 2 2]
>>> print(y_iris[::10])
[0 0 0 0 0 1 1 1 1 1 2 2 2 2 2]
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the Python example above, we used a standard example dataset ‘Iris’, which contains flower petal and sepal dimensions for three different species of iris. We clustered these into three clusters, and compared the obtained clusters to the actual species (target), to see that they match perfectly.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this case, we knew that there were three different clusters (species), and K-Means recognized correctly which ones go together. But, how do we choose a good number of clusters k in general? These kind of questions are quite common in Machine Learning. If we request more clusters, they will be smaller, and therefore the total error (total of distances from points to their assigned clusters) will be smaller. So, is it a good idea just to set a bigger k? We may end with having k = m, that is, each point being its own centroid, with each cluster having only one point. Yes, the total error is 0, but we didn’t get a simpler description of our data, nor is it general enough to cover some new points that may appear. This is called overfitting, and we don’t want that.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A way to deal with this problem is to include some penalty for a larger number of clusters. So, we are now trying to minimize not only the error, but <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">error + penalty</em>. The error will just converge towards zero as we increase the number of clusters, but the penalty will grow. At some points, the gain from adding another cluster will be less than the introduced penalty, and we’ll have the optimal result. A solution that uses<a href="http://www.modelselection.org/bic/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Bayesian Information Criterion</a> (BIC) for this purpose is called X-Means [Pelleg and Moore, 2000].</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another thing we have to define properly is the distance function. Sometimes that’s a straightforward task, a logical one given the nature of data. For points in space, Euclidean distance is an obvious solution, but it may be tricky for features of different ‘units’, for discrete variables, etc. This may require a lot of domain knowledge. Or, we can call Machine Learning for help. We can actually try to learn the distance function. If we have a training set of points that we know how they should be grouped (i.e. points labeled with their clusters), we can use supervised learning techniques to find a good function, and then apply it to our target set that is not yet clustered.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The method used in K-Means, with its two alternating steps resembles an <a href="https://www.ee.washington.edu/techsite/papers/documents/UWEETR-2010-0002.pdf" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Expectation–Maximization</a> (EM) method. Actually, it can be considered a very simple version of EM. However, it should not be confused with the more elaborate EM clustering algorithm even though it shares some of the same principles.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label_title" style="border: 0px; box-sizing: border-box; color: #2557a1; display: inline-block; font-size: 17px; font-weight: 700; line-height: 23px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<div class="embeddable_form-label is-header" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a data-role="preferences_link" href="https://www.toptal.com/machine-learning/clustering-algorithms#" style="border: 0px; box-sizing: border-box; color: #3976cb; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;"></a></div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 864px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h2 id="em-clustering" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
EM Clustering</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, with K-Means clustering each point is assigned to just a single cluster, and a cluster is described only by its centroid. This is not too flexible, as we may have problems with clusters that are overlapping, or ones that are not of circular shape. With <a href="http://docs.rapidminer.com/studio/operators/modeling/segmentation/expectation_maximization_clustering.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">EM Clustering</a>, we can now go a step further and describe each cluster by its centroid (mean), covariance (so that we can have elliptical clusters), and weight (the size of the cluster). The probability that a point belongs to a cluster is now given by a multivariate Gaussian probability distribution (multivariate - depending on multiple variables). That also means that we can calculate the probability of a point being under a Gaussian ‘bell’, i.e. the probability of a point belonging to a cluster.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92523/toptal-blog-image-1463639195822-62ac974368c9d3e4c1501368ae75e321.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We now start the EM procedure by calculating, for each point, the probabilities of it belonging to each of the current clusters (which, again, may be randomly created at the beginning). This is the E-step. If one cluster is a really good candidate for a point, it will have a probability close to one. However, two or more clusters can be acceptable candidates, so the point has a distribution of probabilities over clusters. This property of the algorithm, of points not belonging restricted to one cluster is called “soft clustering”.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The M-step now recalculates the parameters of each cluster, using the assignments of points to the previous set of clusters. To calculate the new mean, covariance and weight of a cluster, each point data is weighted by its probability of belonging to the cluster, as calculated in the previous step.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Alternating these two steps will increase the total log-likelihood until it converges. Again, the maximum may be local, so we can run the algorithm several times to get better clusters.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we now want to determine a single cluster for each point, we may simply choose the most probable one. Having a probability model, we can also use it to generate similar data, that is to sample more points that are similar to the data that we observed.</div>
<h2 id="affinity-propagation" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Affinity Propagation</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://en.wikipedia.org/wiki/Affinity_propagation" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Affinity Propagation</a> (AP) was published by Frey and Dueck in 2007, and is only getting more and more popular due to its simplicity, general applicability, and performance. It is changing its status from state of the art to de facto standard.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92524/toptal-blog-image-1463639242851-65077729f48e9e7f8e0d0ca68cb4a19f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The main drawbacks of K-Means and similar algorithms are having to select the number of clusters, and choosing the initial set of points. Affinity Propagation, instead, takes as input measures of similarity between pairs of data points, and simultaneously considers all data points as potential exemplars. Real-valued messages are exchanged between data points until a high-quality set of exemplars and corresponding clusters gradually emerges.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As an input, the algorithm requires us to provide two sets of data:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Similarities between data points, representing how well-suited a point is to be another one’s exemplar. If there’s no similarity between two points, as in they cannot belong to the same cluster, this similarity can be omitted or set to -Infinity depending on implementation.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Preferences, representing each data point’s suitability to be an exemplar. We may have some a priori information which points could be favored for this role, and so we can represent it through preferences.</div>
</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Both similarities and preferences are often represented through a single matrix, where the values on the main diagonal represent preferences. Matrix representation is good for dense datasets. Where connections between points are sparse, it is more practical not to store the whole n x n matrix in memory, but instead keep a list of similarities to connected points. Behind the scene, ‘exchanging messages between points’ is the same thing as manipulating matrices, and it’s only a matter of perspective and implementation.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92526/toptal-blog-image-1463639329606-7297e0c0f8be49f7f9105830d76848ea.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The algorithm then runs through a number of iterations, until it converges. Each iteration has two message-passing steps:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Calculating responsibilities: Responsibility r(i, k) reflects the accumulated evidence for how well-suited point k is to serve as the exemplar for point i, taking into account other potential exemplars for point i. Responsibility is sent from data point i to candidate exemplar point k.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Calculating availabilities: Availability a(i, k) reflects the accumulated evidence for how appropriate it would be for point i to choose point k as its exemplar, taking into account the support from other points that point k should be an exemplar. Availability is sent from candidate exemplar point k to point i.</div>
</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In order to calculate responsibilities, the algorithm uses original similarities and availabilities calculated in the previous iteration (initially, all availabilities are set to zero). Responsibilities are set to the input similarity between point i and point k as its exemplar, minus the largest of the similarity and availability sum between point i and other candidate exemplars. The logic behind calculating how suitable a point is for an exemplar is that it is favored more if the initial a priori preference was higher, but the responsibility gets lower when there is a similar point that considers itself a good candidate, so there is a ‘competition’ between the two until one is decided in some iteration.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Calculating availabilities, then, uses calculated responsibilities as evidence whether each candidate would make a good exemplar. Availability a(i, k) is set to the self-responsibility r(k, k) plus the sum of the positive responsibilities that candidate exemplar k receives from other points.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Finally, we can have different stopping criteria to terminate the procedure, such as when changes in values fall below some threshold, or the maximum number of iterations is reached. At any point through Affinity Propagation procedure, summing Responsibility (r) and Availability (a) matrices gives us the clustering information we need: for point i, the k with maximum r(i, k) + a(i, k) represents point i’s exemplar. Or, if we just need the set of exemplars, we can scan the main diagonal. If r(i, i) + a(i, i) > 0, point i is an exemplar.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We’ve seen that with K-Means and similar algorithms, deciding the number of clusters can be tricky. With AP, we don’t have to explicitly specify it, but it may still need some tuning if we obtain either more or less clusters than we may find optimal. Luckily, just by adjusting the preferences we can lower or raise the number of clusters. Setting preferences to a higher value will lead to more clusters, as each point is ‘more certain’ of its suitability to be an exemplar and is therefore harder to ‘beat’ and include it under some other point’s ‘domination’. Conversely, setting lower preferences will result in having less clusters; as if they’re saying “no, no, please, you’re a better exemplar, I’ll join your cluster”. As a general rule, we may set all preferences to the median similarity for a medium to large number of clusters, or to the lowest similarity for a moderate number of clusters. However, a couple of runs with adjusting preferences may be needed to get the result that exactly suits our needs.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Hierarchical Affinity Propagation is also worth mentioning, as a variant of the algorithm that deals with quadratic complexity by splitting the dataset into a couple of subsets, clustering them separately, and then performing the second level of clustering.</div>
<h2 id="in-the-end" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In The End…</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There’s a whole range of clustering algorithms, each one with its pros and cons regarding what type of data they with, time complexity, weaknesses, and so on. To mention more algorithms, for example there’s Hierarchical Agglomerative Clustering (or Linkage Clustering), good for when we don’t necessarily have circular (or hyper-spherical) clusters, and don’t know the number of clusters in advance. It starts with each point being a separate cluster, and works by joining two closest clusters in each step until everything is in one big cluster.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With <a href="http://nlp.stanford.edu/IR-book/html/htmledition/hierarchical-agglomerative-clustering-1.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Hierarchical Agglomerative Clustering</a>, we can easily decide the number of clusters afterwards by cutting the dendrogram (tree diagram) horizontally where we find suitable. It is also repeatable (always gives the same answer for the same dataset), but is also of a higher complexity (quadratic).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92525/toptal-blog-image-1463639301381-ac46f23d267a0150ea7f3804b456a802.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Then, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is also an algorithm worth mentioning. It groups points that are closely packed together, expanding clusters in any direction where there are nearby points, thus dealing with different shapes of clusters.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
These algorithms deserve an article of their own, and we can explore them in a separate post later on.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It <a href="https://www.toptal.com/machine-learning" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">takes experience</a> with some trial and error to know when to use one algorithm or the other. Luckily, we have a range of implementations in different programming languages, so trying them out only requires a little willingness to play.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <span style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/resume/lovro-iliasich" style="border: 0px; box-sizing: border-box; color: #103d77; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">LOVRO ILIASSICH</a></span><span style="font-size: 1.2em; line-height: 1.5em;">, a <a href="https://www.toptal.com/machine-learning/clustering-algorithms" target="_blank">Topta</a>l<a href="https://www.toptal.com/java" target="_blank"> Java developer.</a></span></div>
Anonymousnoreply@blogger.com2tag:blogger.com,1999:blog-8706416286757464743.post-83647696485598310242016-06-02T20:05:00.003+07:002016-06-02T20:05:52.956+07:00Rise Of Automated Trading: Machines Trading S&P 500<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Nowadays, more than 60 percent of trading activities with different assets (such as stocks, index futures, commodities) are not made by “human being” traders anymore, instead relying on automated trading. There are specialized programs based on particular algorithms that automatically buy and sell assets over different markets, meant to achieve a positive return in the long run.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this article, I’m going to show you how to predict, with good accuracy, how the next trade should be placed to get a positive gain. For this example, as the underlying asset to trade, I selected the S&P 500 index, the weighted average of 500 US companies with bigger capitalization. A very simple strategy to implement is to buy the S&P 500 index when Wall Street Exchange starts trading, at 9:30 AM, and selling it at the closing session at 4:00 PM Eastern Time. If the closing price of the index is higher than the opening price, there is a positive gain, whereas a negative gain would be achieved if the closing price is lower than the opening price. So the question is: how do we know if the trading session will end up with a closing price higher than opening price? <a href="https://www.toptal.com/machine-learning" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Machine Learning</a> is a powerful tool to achieve such a complex task, and it can be a useful tool to support us with the trading decision.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Machine Learning is the new frontier of many useful real life applications. Financial trading is one of these, and it’s used very often in this sector. An important concept about <a href="https://www.toptal.com/machine-learning" target="_blank">Machine Learning</a> is that we do not need to write code for every kind of possible rules, such as pattern recognition. This is because every model associated with Machine Learning learns from the data itself, and then can be later used to predict unseen new data.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Machine Learning is the new frontier of many useful real life applications" src="https://assets.toptal.io/uploads/blog/image/92573/toptal-blog-image-1464608547350-1ed82c7f5be8c89f1751be0b513a1e99.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Disclaimer</span>: The purpose of this article is to show how to train Machine Learning methods, and in the provided code examples not every function is explained. This article is not intended to let one copy and paste all the code and run the same provided tests, as some details are missing that were out of the scope the article. Also, base knowledge of Python is required. The main intention of the article is to show an example of how machine learning may be effective to predict buys and sells in the financial sector. However, trade with real money means to have many other skills, such as money management and risk management. This article is just a small piece of the “big picture”.</div>
<h2 id="building-your-first-financial-data-automated-trading-program" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Building Your First Financial Data Automated Trading Program</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, you want to create your first program to analyze financial data and predict the right trade? Let me show you how. I will be using Python for Machine Learning code, and we will be using historical data from <a href="https://pypi.python.org/pypi/yahoo-finance" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Yahoo Finance service</a>. As mentioned before, historical data is necessary to train the model before making our predictions.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To begin, we need to install:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Python, and in particular I suggest using <a href="http://jupyter.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">IPython notebook</a>.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://pypi.python.org/pypi/yahoo-finance" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Yahoo Finance Python package</a> (the exact name is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">yahoo-finance</code>) through the terminal command: <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">pip install yahoo-finance</code>.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">A free trial version of Machine Learning package called <a href="https://dato.com/download/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">GraphLab</a>. Feel free to check the useful <a href="https://dato.com/products/create/docs/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">documentation of that library</a>.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that only a part of GraphLab is open source, the <a href="https://pypi.python.org/pypi/SFrame" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">SFrame</a>, so to use the entire library we need a license. There is a 30 day free license and a non-commercial license for students or those one participating in Kaggle competitions. From my point of view, GraphLab Create is a very intuitive and easy to use library to analyze data and train Machine Learning models.</div>
<h3 id="digging-in-the-python-code" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Digging in the Python Code</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s dig in with some Python code to see how to download financial data from the Internet. I suggest using IPython notebook to test the following code, because IPython has many advantages compared to a traditional IDE, especially when we need to combine source code, execution code, table data and charts together on the same document. For a brief explanation to use IPython notebook, please look at the <a href="http://opentechschool.github.io/python-data-intro/core/notebook.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Introduction to IPython Notebook</a> article.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, let’s create a new IPython notebook and write some code to download historical prices of S&P 500 index. Note, if you prefer to use other tools, you can start with a new Python project in your preferred IDE.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> graphlab <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">as</span> gl
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">from</span> __future__ <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> division
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">from</span> datetime <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> datetime
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">from</span> yahoo_finance <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> Share
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># download historical prices of S&P 500 index</span>
today = datetime.strftime(datetime.today(), <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"%Y-%m-%d"</span>)
stock = Share(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'^GSPC'</span>) <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># ^GSPC is the Yahoo finance symbol to refer S&P 500 index</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># we gather historical quotes from 2001-01-01 up to today</span>
hist_quotes = stock.get_historical(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2001-01-01'</span>, today)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># here is how a row looks like</span>
hist_quotes[<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>]
{<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Adj_Close'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2091.580078'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Close'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2091.580078'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Date'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2016-04-22'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'High'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2094.320068'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Low'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2081.199951'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Open'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'2091.48999'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Symbol'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'%5eGSPC'</span>,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Volume'</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'3790580000'</span>}
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">hist_quotes</code> is a list of dictionaries, and each dictionary object is a trading day with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Open</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">High</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Low</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Close</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Adj_close</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Volume</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Symbol</code> and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Date</code> values. During each trading day, the price usually changes starting from the opening price <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Open</code> to the closing price <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Close</code>, and hitting a maximum and a minimum value <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">High</code> and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Low</code>. We need to read through it and create lists of each of the most relevant data. Also, data must be ordered by the most recent values at first, so we need to reverse it:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">l_date = []
l_open = []
l_high = []
l_low = []
l_close = []
l_volume = []
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># reverse the list</span>
hist_quotes.reverse()
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> quotes <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span> hist_quotes:
l_date.append(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Date'</span>])
l_open.append(float(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Open'</span>]))
l_high.append(float(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'High'</span>]))
l_low.append(float(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Low'</span>]))
l_close.append(float(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Close'</span>]))
l_volume.append(int(quotes[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Volume'</span>]))
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can pack all downloaded quotes into an <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">SFrame</code> object, which is a highly scalable column based data frame, and it is compressed. One of the advantages is that it can also be larger than the amount of RAM because it is disk-backed. You can check the <a href="https://dato.com/products/create/docs/generated/graphlab.SFrame.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">documentation</a> to learn more about SFrame.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, let’s store and then check the historical data:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">qq = gl.SFrame({<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span> : l_date,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span> : l_open,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'high'</span> : l_high,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'low'</span> : l_low,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span> : l_close,
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'volume'</span> : l_volume})
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># datetime is a string, so convert into datetime object</span>
qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>] = qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>].apply(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lambda</span> x:datetime.strptime(x, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'%Y-%m-%d'</span>))
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># just to check if data is sorted in ascending mode</span>
qq.head(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>)
</code></pre>
<table class="content_table" style="border-collapse: collapse; border-spacing: 0px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: 0px 0px 18px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline; width: 100%px;"><tbody style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<tr class="content_table-row_title" style="background-color: #2557a1; border: 0px; box-sizing: border-box; color: white; font-weight: 700; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">close</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">datetime</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">high</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">low</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">open</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">volume</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1283.27</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2001-01-02 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1320.28</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1276.05</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1320.28</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1129400000</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1347.56</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2001-01-03 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1347.76</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1274.62</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1283.27</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1880700000</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1333.34</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2001-01-04 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1350.24</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1329.14</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1347.56</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2131000000</td></tr>
</tbody></table>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now we can save data to disk with the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">SFrame</code> method <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">save</code>, as follows:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">qq.save(“SP500_daily.bin”)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># once data is saved, we can use the following instruction to retrieve it </span>
qq = gl.SFrame(“SP500_daily.bin/”)
</code></pre>
<h3 id="lets-see-what-the-sp-500-looks-like" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s See What the S&P 500 Looks Like</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To see how the loaded S&P 500 data will look like, we can use the following code:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> matplotlib.pyplot <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">as</span> plt
%matplotlib inline <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># only for those who are using IPython notebook</span>
plt.plot(qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>])
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The output of the code is the following graph:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92560/toptal-blog-image-1464605080285-b49d67f47e1cdad0bd119b21f8e4f82b.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="training-some-machine-learning-models" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training Some Machine Learning Models</h2>
<h3 id="adding-outcome" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Adding Outcome</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As I stated in the introductory part of this article, the goal of each model is to predict if the closing price will be higher than the opening price. Hence, in that case, we can achieve a positive return when buying the underlying asset. So, we need to add an <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">outcome</code> column on our data which will be the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">target</code> or <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predicted</code> variable. Every row of this new column will be:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> for an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up day</em> with a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Closing</code> price higher than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Opening</code> price.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code> for a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Down day</em> with a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Closing</code> price lower than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Opening</code> price.</li>
</ul>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># add the outcome variable, 1 if the trading session was positive (close>open), 0 otherwise</span>
qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>] = qq.apply(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lambda</span> x: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> x[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] > x[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>] <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">else</span> -<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># we also need to add three new columns ‘ho’ ‘lo’ and ‘gain’</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># they will be useful to backtest the model, later</span>
qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'ho'</span>] = qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'high'</span>] - qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># distance between Highest and Opening price</span>
qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'lo'</span>] = qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'low'</span>] - qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># distance between Lowest and Opening price</span>
qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>] = qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] - qq[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>]
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since we need to assess some days before the last trading day, we need to <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lag</em> data by one or more days. For that kind of <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lagging</em> operation, we need another object from GraphLab package called <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">TimeSeries</code>. <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">TimeSeries</code> has a method <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">shift</code> that lags data by a certain number of rows.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ts = gl.TimeSeries(qq, index=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># add the outcome variable, 1 if the bar was positive (close>open), 0 otherwise</span>
ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>] = ts.apply(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lambda</span> x: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> x[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] > x[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>] <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">else</span> -<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># GENERATE SOME LAGGED TIMESERIES</span>
ts_1 = ts.shift(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>) <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># by 1 day</span>
ts_2 = ts.shift(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>) <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># by 2 days</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># ...etc....</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># it's an arbitrary decision how many days of lag are needed to create a good forecaster, so</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># everyone can experiment by his own decision</span>
</code></pre>
<h3 id="adding-predictors" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Adding Predictors</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Predictors</em> are a set of feature variables that must be chosen to train the model and predict our <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">outcome</em>. So, forecasting factor choice is crucial, if not the most important, component of the forecaster.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Just to name a few examples, a factor to consider may be if today’s close is higher than yesterday’s close, and that might be extended with two previous days’ close, etc. A similar choice can be translated with the following code:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'feat1'</span>] = ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] > ts_1[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>]
ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'feat2'</span>] = ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] > ts_2[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>]
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As shown above, I’ve added two new features columns, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">feat1</code> and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">feat2</code> on our data set (<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">ts</code>) containing <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">1</code> if the comparison is true and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0</code> otherwise.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article is intended to give an example of Machine Learning applied to the Financial sector. I prefer to focus on how Machine Learning models may be used with financial data, and we will not go into detail regarding how to choose the right factors to train the models. It is too exhaustive to explain why certain factors are used in respect to others, due to a considerable increase in complexity. My job research is to study many hypotheses of choosing factors to create a good predictor. So for a start, I suggest you experiment with lots of different combinations of factors, to see if they may increase the accuracy of the model.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># add_features is a helper function, which is out of the scope of this article,</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># and it returns a tuple with:</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># ts: a timeseries object with, in addition to the already included columns, also lagged columns</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># as well as some features added to train the model, as shown above with feat1 and feat2 examples</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># l_features: a list with all features used to train Classifier models</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># l_lr_features: a list all features used to train Linear Regression models</span>
ts, l_features, l_lr_features = add_features(ts)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># add the gain column, for trading operations with LONG only positions. </span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># The gain is the difference between Closing price - Opening price</span>
ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>] = ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>] - ts[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>]
ratio = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.8</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># 80% of training set and 20% of testing set</span>
training = ts.to_sframe()[<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>:round(len(ts)*ratio)]
testing = ts.to_sframe()[round(len(ts)*ratio):]
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Building Your First Financial Data Automated Trading Program" src="https://assets.toptal.io/uploads/blog/image/92572/toptal-blog-image-1464608096751-87f9c1a11bfe89bc96bcb6b929a29502.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="training-a-decision-tree-model" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training a Decision Tree Model</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
GraphLab Create has a very clean interface to implement Machine Learning models. Each model has a method <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">create</code> used to fit the model with a training data set. Typical parameters are:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">training</code> - it is a training set containing feature columns and a target column.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">target</code> - it is the name of the column containing the target variable.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">validation_set</code> - it is a dataset for monitoring the model’s generalization performance. In our case, we have no <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">validation_set</code>.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">features</code> - it is a list of columns names of features used for training the model.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">verbose</code> - if <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">true</code>, print progress information during training.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Whereas other parameters are typical of the model itself, such as:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">max_depth</code> - it is the maximum depth of a tree.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With the following code we build our decision tree:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">max_tree_depth = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">6</span>
decision_tree = gl.decision_tree_classifier.create(training, validation_set=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>,
target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>, features=l_features,
max_depth=max_tree_depth, verbose=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False</span>)
</code></pre>
<h3 id="measuring-performance-of-fitted-model" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Measuring Performance of Fitted Model</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Accuracy</span> is an important metric to evaluate the goodness of the forecaster. It is the number of correct predictions divided by the number of total data points. Since the model is fitted with training data, the accuracy evaluated with the training set is better than the one obtained with a test set.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Precision</span> is the fraction of positive predictions that are positive. We need precision to be a number closer to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">1</code>, to achieve a “perfect” win-rate. Our <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">decision_tree</code>, as another classifier from GraphLab Create package, has its method <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">evaluate</code> to get many important metrics of the fitted model.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Recall</span> quantifies the ability of a classifier to predict positive examples. Recall can be interpreted as the probability that a randomly selected positive example is correctly identified by the classifier. We need that precision would be a number closer to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">1</code>, to achieve a “perfect” win-rate.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The following code will show the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">accuracy</em> of the fitted model both with training set and testing set:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">decision_tree.evaluate(training)[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'accuracy'</span>], decision_tree.evaluate(testing)[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'accuracy'</span>]
(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6077348066298343</span>, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.577373211963589</span>)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As shown above, the accuracy of the model with the test set is about 57 percent, which is somehow better than tossing a coin (50 percent).</div>
<h3 id="predicting-data" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Predicting Data</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
GraphLab Create has the same interface to predict data from different fitted models. We will use the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predict</code>method, which needs a test set to predict the target variable, in our case <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">outcome</code>. Now, we can predict data from the testing set:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">predictions = decision_tree.predict(testing)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># and we add the predictions column in testing set</span>
testing[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'predictions'</span>] = predictions
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># let's see the first 10 predictions, compared to real values (outcome column)</span>
testing[[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'predictions'</span>]].head(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10</span>)
</code></pre>
<table class="content_table" style="border-collapse: collapse; border-spacing: 0px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: 0px 0px 18px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline; width: 100%px;"><tbody style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<tr class="content_table-row_title" style="background-color: #2557a1; border: 0px; box-sizing: border-box; color: white; font-weight: 700; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">datetime</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">outcome</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">predictions</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-05 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-08 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-09 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-10 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-11 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-12 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-15 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-16 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-17 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">2013-04-18 00:00:00</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">-1</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">1</td></tr>
</tbody></table>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False positives</span> are cases where the model predicts a positive outcome whereas the real outcome from the testing set is negative. Vice versa, <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False negatives</span> are cases where the model predicts a negative outcome where the real outcome from the test set is positive.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Our trading strategy waits for a positively predicted outcome to buy S&P 500 at the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Opening</code> price, and sell it at the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Closing</code> price, so our hope is to have the lowest False positives rate to avoid losses. In other words, we expect our model would have the <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">highest precision</span> rate.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As we can see, there are two false negatives (at 2013-04-10 and 2013-04-11) and two false positives (at 2013-04-15 and 2013-04-18) within the first ten predicted values of the testing set.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With a simple calculation, considering this small set of ten predictions:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">accuracy = 6/10 = 0.6 or 60%</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">precision =3/5 = 0.6 or 60%</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">recall = 3/5 = 0.6 or 60%</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that usually the numbers above are different from each other, but in this case they are the same.</div>
<h3 id="backtesting-the-model" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Backtesting the Model</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We now simulate how the model would trade using its predicted values. If the predicted outcome is equal to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code>, it means that we expect an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up day</em>. With an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up day</em> we buy the index at the beginning of the session, and sell the index at the end of the session during the same day. Conversely, if the predicted outcome is equal to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code> we expect a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Down day</em>, so we will not trade during that day.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Profit and Loss (<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">pnl</code>) for a complete daily trade, also called <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">round turn</em>, in this example is given by:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">pnl = Close - Open</code> (for each trading day)</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With the code shown below, I call the helper function <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">plot_equity_chart</code> to create a chart with the curve of cumulative gains (equity curve). Without going too deep, it simply gets a series of profit and loss values and calculates the series of cumulative sums to plot.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">pnl = testing[testing[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'predictions'</span>] == <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>][<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>] <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># the gain column contains (Close - Open) values</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># I have written a simple helper function to plot the result of all the trades applied to the</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># testing set and represent the total return expressed by the index basis points</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># (not expressed in dollars $)</span>
plot_equity_chart(pnl,<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Decision tree model'</span>)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92561/toptal-blog-image-1464605314055-589acecd4ed5805732f274990f18fb05.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 1.843504
Sharpe is 1.972835
Round turns 511
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here, Sharpe is the Annual <a href="http://www.investopedia.com/articles/07/sharpe_ratio.asp" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Sharpe</a> ratio, an important indicator of the goodness of the trading model. Considering trades expressed day by day<img alt="" src="https://assets.toptal.io/uploads/blog/image/92574/toptal-blog-image-1464642133921-e3038141cb448d61c38a70c730b63d5b.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" />whereas <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">mean</code> is the mean of the list of profit and loss, and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">sd</code> is the standard deviation. For simplicity in the formula depicted above, I have considered a risk-free return equal to 0.</div>
<h2 id="some-basics-about-trading" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Some Basics About Trading</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Trading the index requires buying an asset, which is directly derived from the index. Many brokers replicate the S&P 500 index with a derivative product called <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">CFD</span> (Contract for difference), which is an agreement between two parties to exchange the difference between the opening price and closing price of a contract.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Example</span>: Buy 1 CFD S&P 500 at <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Open</code> (value is 2000), sell it at <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Close</code> of the day (value is 2020). The difference, hence the gain, is 20 points. If each point has a value of $25:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Gross Gain</span> is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">20 points x $25 = $500</code> with 1 CFD contract.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s say that the broker keeps a slippage of 0.6 points for its own revenue:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Net gain</span> is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">(20 - 0.6) points x $25 = $485</code>.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another important aspect to consider is to avoid significant losses within a trade. They may happen whenever the predicted outcome is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> but the real outcome value is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code>, so it is a <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false positive</span>. In that case, the ending session turns out to be a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Down day</em> with a closing price lower than the opening, and we get a loss.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">stop loss order</span> must be placed to protect against a maximum loss we would tolerate within a trade, and such an order is triggered whenever the price of the asset goes below a fixed value we have set before.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we look at the time series downloaded from Yahoo Finance at the beginning of this article, every day has a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Low</code> price which is the lowest price reached during that day. If we set a stop level of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-3</code> points far from the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Opening</code> price, and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Low - Open = -5</code> the stop order will be triggered, and the opened position will be closed with a loss of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-3</code> points instead of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-5</code>. This is a simple method to reduce the risk. The following code represents my helper function to simulate a trade with a stop level:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># This is a helper function to trade 1 bar (for example 1 day) with a Buy order at opening session</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># and a Sell order at closing session. To protect against adverse movements of the price, a STOP order</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># will limit the loss to the stop level (stop parameter must be a negative number)</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># each bar must contains the following attributes: </span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># Open, High, Low, Close prices as well as gain = Close - Open and lo = Low - Open</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">trade_with_stop</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(bar, slippage = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>, stop=None)</span>:</span>
<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"""
Given a bar, with a gain obtained by the closing price - opening price
it applies a stop limit order to limit a negative loss
If stop is equal to None, then it returns bar['gain']
"""</span>
bar[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>] = bar[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>] - slippage
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> stop<><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>:
real_stop = stop - slippage
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> bar[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'lo'</span>]<=stop:
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> real_stop
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># stop == None </span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> bar[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>]
</code></pre>
<h3 id="trading-costs" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Trading Costs</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Transaction costs are expenses incurred when buying or selling securities. Transaction costs include brokers’ commissions and spreads (the difference between the price the dealer paid for a security and the price the buyer pays), and they need to be considered if we want to backtest our strategy, similarly to a real scenario. Slippage in the trading of stocks often occurs when there is a change in spread. In this example and for the next ongoing simulations, trading costs are fixed as:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Slippage = 0.6 points</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Commission = 1$ for each trade (one round turn will cost 2$)</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Just to write some numbers, if our gross gain were 10 points, 1 point = $25, so $250 including trading costs, our net gain would be <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">(10 - 0.6)*$25 - 2 = $233</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The following code shows a simulation of the previous trading strategy with a stop loss of -3 points. The blue curve is the curve of cumulative returns. The only costs accounted for are slippage (0.6 points), and the result is expressed in basis points (the same base unit of S&P 500 values downloaded from Yahoo Finance).</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">SLIPPAGE = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6</span>
STOP = -<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>
trades = testing[testing[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'predictions'</span>] == <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>][(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'ho'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'lo'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>)]
trades[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'pnl'</span>] = trades.apply(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">lambda</span> x: trade_with_stop(x, slippage=SLIPPAGE, stop=STOP))
plot_equity_chart(trades[<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'pnl'</span>],<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Decision tree model'</span>)
print(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Slippage is %s, STOP level at %s"</span> % (SLIPPAGE, STOP))
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92563/toptal-blog-image-1464605547032-19dc2d1d8db4eeef6fbe068792f23528.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 2.162171
Sharpe is 3.502897
Round turns 511
Slippage is 0.6
STOP level at -3
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The following code is used to make predictions in a slightly different way. Please pay attention to the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predict</code> method which is called with an additional parameter <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">output_type = “probability”</code>. This parameter is used to return probabilities of predicted values instead of their class prediction (<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> for a positively predicted outcome, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code> for a negatively predicted outcome). A probability greater than or equal to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.5</code> is associated with a predicted value <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> and a probability value less than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.5</code> is related to a predicted value of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code>. The higher that probability is, the more chance we have to predict a real <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up Day</em>.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">predictions_prob = decision_tree.predict(testing, output_type = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'probability'</span>)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># predictions_prob will contain probabilities instead of the predicted class (-1 or +1)</span>
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now we backtest the model with a helper function called <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">backtest_ml_model</code> which calculates the series of cumulative returns including slippage and commissions, and plots their values. For brevity, without explaining thoroughly <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">backtest_ml_model</code> function, the important detail to highlight is that instead of filtering those days with a predicted <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">outcome = 1</code> as we did in the previous example, now we filter those <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predictions_prob</code>equal to or greater than a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">threshold = 0.5</code>, as following:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">trades = testing[predictions_prob>=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.5</span>][(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'datetime'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'ho'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'lo'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'open'</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'close'</span>)]
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Remember that <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Net gain</em> of each trading day is: <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Net gain = (Gross gain - SLIPPAGE) * MULT - 2 * COMMISSION</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another important metric used to evaluate the goodness of a trading strategy is the <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Maximum Drawdown</span>. In general, it measures the largest single drop from peak to the bottom, in the value of an invested portfolio. In our case, it is the most significant drop from peak to bottom of the equity curve (we have just one asset in our portfolio, S&P 500). So given an <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">SArray</code> of profit and loss <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">pnl</code>, we calculate the drawdown as:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">drawdown = pnl - pnl.cumulative_max()
max_drawdown = min(drawdown)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Inside the helper function <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">backtest_summary</code> is calculated:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Maximum drawdown (in dollars) as shown above.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Accuracy, with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Graphlab.evaluation</code> method.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Precision, with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Graphlab.evaluation</code> method.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Recall, with<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Graphlab.evaluation</code> method.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Putting it all together, the following example shows the equity curve representing cumulative returns of the model strategy, with all values expressed in dollars.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">model = decision_tree
predictions_prob = model.predict(testing, output_type=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"probability"</span>)
THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.5</span>
bt_1_1 = backtest_ml_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>,
threshold=THRESHOLD, STOP=-<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>,
MULT=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">25</span>, SLIPPAGE=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6</span>, COMMISSION=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>, plot_title=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'DecisionTree'</span>)
backtest_summary(bt_1_1)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92564/toptal-blog-image-1464605578712-afe8dcb1999380e31198a2eed4d53a73.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 54.054286
Sharpe is 3.502897
Round turns 511
Name: DecisionTree
Accuracy: 0.577373211964
Precision: 0.587084148728
Recall: 0.724637681159
Max Drawdown: -1769.00025
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To increase the precision of forecasted values, instead of a standard probability of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.5</code> (50 percent) we choose a higher threshold value, to be more confident that the model predicts an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up day</em>.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.55</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># it’s the minimum threshold to predict an Up day so hopefully a good day to trade</span>
bt_1_2 = backtest_ml_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>,
threshold=THRESHOLD, STOP=-<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>,
MULT=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">25</span>, SLIPPAGE=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6</span>, COMMISSION=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>, plot_title=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'DecisionTree'</span>)
backtest_summary(bt_1_2)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92565/toptal-blog-image-1464605609785-b559def89ca2673254df1c5feaca7e5f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 118.244689
Sharpe is 6.523478
Round turns 234
Name: DecisionTree
Accuracy: 0.560468140442
Precision: 0.662393162393
Recall: 0.374396135266
Max Drawdown: -1769.00025
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As we can see by the chart above, the equity curve is much better than before (Sharpe is 6.5 instead of 3.5), even with fewer round turns.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
From this point on, we will consider all next models with a threshold higher than a standard value.</div>
<h2 id="training-a-logistic-classifier" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training a Logistic Classifier</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can apply our research, as we did previously with the decision tree, into a Logistic Classifier model. GraphLab Create has the same interface with Logistic Classifier object, and we will call the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">create</code> method to build our model with the same list of parameters. Moreover, we prefer to predict the probability vector instead of the predicted class vector (composed of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> for a positive outcome, and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code> for a negative outcome), so we would have a threshold greater than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.5</code> to achieve a better precision in our forecasting.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">model = gl.logistic_classifier.create(training, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>, features=l_features,
validation_set=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>, verbose=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False</span>)
predictions_prob = model.predict(testing, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'probability'</span>)
THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6</span>
bt_2_2 = backtest_ml_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>,
threshold=THRESHOLD, STOP=-<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>, plot_title=model.name())
backtest_summary(bt_2_2)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92566/toptal-blog-image-1464605635951-618337931bfd72a178140090f3981633.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 112.704215
Sharpe is 6.447859
Round turns 426
Name: LogisticClassifier
Accuracy: 0.638491547464
Precision: 0.659624413146
Recall: 0.678743961353
Max Drawdown: -1769.00025
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this case, there is a summary very similar to Decision Tree. After all, both models are classifiers, they only predict a class of binary outcomes (<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code>).</div>
<h2 id="training-a-linear-regression-model" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training a Linear Regression Model</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The main difference of this model is that it deals with continuous values instead of binary classes, as mentioned before. We don’t have to train the model with a target variable equal to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">+1</code> for <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up days</em> and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-1</code>for <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Down days</em>, our target must be a continuous variable. Since we want to predict a positive gain, or in other words a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Closing</em> price higher than the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Opening</em> price, now target must be the gain column of our training set. Also, the list of features must be composed of continuous values, such as the previous <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Open</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Close</code>, etc.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For brevity, I won’t go into the details of how to select the right features, as this is beyond the scope of this article, which is more inclined to show how we should apply different Machine Learning models over a data set. The list of parameters passed to the create method are:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">training</code> - it is a training set containing feature columns and a target column.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">target</code> - it is the name of the column containing the target variable.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">validation_set</code> - it is a dataset for monitoring the model’s generalization performance. In our case, we have no <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">validation_set</code>.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">features</code> - it is a list of column names of features used for training the model, for this model we will use another set respect to the Classifier models.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">verbose</code> - if <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">true</code>, it will print progress information during training.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">max_iterations</code> - it is the maximum number of allowed passes through the data. More passes over the data can result in a more accurately trained model.</li>
</ul>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">model = gl.linear_regression.create(training, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>, features = l_lr_features,
validation_set=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>, verbose=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False</span>, max_iterations=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100</span>)
predictions = model.predict(testing)
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># a linear regression model, predict continuous values, so we need to make an estimation of their</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># probabilities of success and normalize all values in order to have a vector of probabilities</span>
predictions_max, predictions_min = max(predictions), min(predictions)
predictions_prob = (predictions - predictions_min)/(predictions_max - predictions_min)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So far, we have predictions which is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">SArray</code> of predicted gains, whereas <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predictions_prob</code> is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">SArray</code> with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predictions</code> values normalized. To have a good accuracy and a certain number of round turns, comparable with previous models, I’ve chosen a threshold value of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.4</code>. For a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">predictions_prob</code> less than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.4</code>, the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">backtest_linear_model</code> helper function will not open a trade because a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Down day</em> is expected. Otherwise, a trade will be opened.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.4</span>
bt_3_2 = backtest_linear_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'gain'</span>, threshold=THRESHOLD,
STOP = -<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>, plot_title=model.name())
backtest_summary(bt_3_2)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92567/toptal-blog-image-1464605680782-2106229d6a21e3b92150c36324a7972f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 138.868280
Sharpe is 7.650187
Round turns 319
Name: LinearRegression
Accuracy: 0.631989596879
Precision: 0.705329153605
Recall: 0.54347826087
Max Drawdown: -1769.00025
</code></pre>
<h2 id="training-a-boosted-tree" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training a Boosted Tree</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As we previously did training a decision tree, now we are going to train a boosted tree classifier with the same parameters used for other classifier models. In addition, we set the number of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">max_iterations = 12</code> in order to increase the maximum number of iterations for boosting. Each iteration results in the creation of an extra tree. We also set a higher value of threshold than <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">0.5</code> to increase precision.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">model = gl.boosted_trees_classifier.create(training, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>, features=l_features,
validation_set=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>, max_iterations=<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">12</span>, verbose=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False</span>)
predictions_prob = model.predict(testing, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'probability'</span>)
THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.7</span>
bt_4_2 = backtest_ml_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>,
threshold=THRESHOLD, STOP=-<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>, plot_title=model.name())
backtest_summary(bt_4_2)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92568/toptal-blog-image-1464605710876-dc3d452e21bec5a1d48062271ede510f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 112.002338
Sharpe is 6.341981
Round turns 214
Name: BoostedTreesClassifier
Accuracy: 0.563068920676
Precision: 0.682242990654
Recall: 0.352657004831
Max Drawdown: -1769.00025
</code></pre>
<h2 id="training-a-random-forest" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Training a Random Forest</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is our last trained model, a Random Forest Classifier, composed by an ensemble of decision trees. The maximum number of trees to use in the model is set to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">num_trees = 10</code>, to avoid too much complexity and overfitting.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-python hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">model = gl.random_forest_classifier.create(training, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>, features=l_features,
validation_set=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">None</span>, verbose=<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">False</span>, num_trees = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">10</span>)
predictions_prob = model.predict(testing, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'probability'</span>)
THRESHOLD = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0.6</span>
bt_5_2 = backtest_ml_model(testing, predictions_prob, target=<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'outcome'</span>,
threshold=THRESHOLD, STOP=-<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>, plot_title=model.name())
backtest_summary(bt_5_2)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92569/toptal-blog-image-1464605738869-8b77e86a729d074fec4fb2fc83ba27aa.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 114.786962
sharpe is 6.384243
Round turns 311
Name: RandomForestClassifier
Accuracy: 0.598179453836
Precision: 0.668810289389
Recall: 0.502415458937
Max Drawdown: -1769.00025
</code></pre>
<h2 id="collecting-all-the-models-together" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Collecting All the Models Together</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now we can join all the strategies together and see the overall result. It’s interesting to see the summary of all Machine Learning models, sorted by their precision.</div>
<table class="content_table" style="border-collapse: collapse; border-spacing: 0px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: 0px 0px 18px; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline; width: 100%px;"><tbody style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<tr class="content_table-row_title" style="background-color: #2557a1; border: 0px; box-sizing: border-box; color: white; font-weight: 700; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">name</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">accuracy</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">precision</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">round turns</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">sharpe</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">LinearRegression</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.63</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.71</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">319</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">7.65</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">BoostedTreesClassifier</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.56</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.68</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">214</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">6.34</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">RandomForestClassifier</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.60</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.67</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">311</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">6.38</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">DecisionTree</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.56</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.66</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">234</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">6.52</td></tr>
<tr class="content_table-row" style="background-color: #dde7f3; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">LogisticClassifier</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.64</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">0.66</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">426</td><td style="border: 2px solid rgb(255, 255, 255); box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 5px 10px; vertical-align: baseline;">6.45</td></tr>
</tbody></table>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we collect all the profit and loss for each one of the previous models in the array <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">pnl</code>, the following chart depicts the equity curve obtained by the sum of each profit and loss, day by day.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92570/toptal-blog-image-1464605784703-991ce9a9379e25b88c0abd7a5003951c.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Mean of PnL is 119.446463
Sharpe is 6.685744
Round turns 1504
First trading day 2013-04-09
Last trading day 2016-04-22
Total return 179647
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Just to give some numbers, with about 3 years of trading, all models have a total gain of about 180,000 dollars. The maximum exposition is 5 CFD contracts in the market, but to reduce the risk they all are closed at the end of each day, so overnight positions are not allowed.</div>
<h2 id="statistics-of-the-aggregation-of-all-models-together" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Statistics of the Aggregation of All Models Together</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since each model can open a trade, but we added 5 concurrent models together, during the same day there could be from 1 contract up to 5 CFD contracts. <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">If all models agree to open trades during the same day, there is a high chance to have an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Up day</em> predicted.</span> Moreover, we can group by the number of models that open a trade at the same time during the opening session of the day. Then we evaluate precision as a function of the number of concurrent models.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92571/toptal-blog-image-1464605820045-e2a0d17088704a7a855b95c32c13a879.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As we can see by the chart depicted above, the precision gets better as the number of models do agree to open a trade. The more models agree, the more precision we get. For instance, with 5 models triggered the same day, the chance to predict an Up day is greater than 85%.</div>
<h2 id="conclusion" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Conclusion</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Even in the financial world, Machine Learning is welcomed as a powerful instrument to learn from data and give us great forecasting tools. Each model shows different values of accuracy and precision, but in general, all models can be aggregated to achieve a better result than each one of them taken singularly. GraphLab Create is a great library, easy to use, scalable and able to manage Big Data very quickly. It implements different scientific and forecasting models, and there is a free license for students and Kaggle competitions.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<i>This post originally appeared in <a href="https://www.toptal.com/machine-learning/s-p-500-automated-trading" target="_blank">Toptal Engineering blog</a></i></div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-25583465729659131612016-05-27T10:30:00.000+07:002016-05-27T10:30:35.580+07:00HTTP Request Testing: A Developer's Survival Tool<h2 id="what-to-do-when-a-testing-suite-isnt-feasible" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What To Do When A Testing Suite Isn’t Feasible</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are times that we – programmers and/or our clients – have limited resources with which to write both the expected deliverable and the automated tests for that deliverable. When the application is small enough you can cut corners and skip tests because you remember (mostly) what happens elsewhere in the code when you add a feature, fix a bug, or refactor. That said, we won’t always work with small applications, plus, they tend to get bigger and more complex over time. This makes manual testing difficult and super annoying.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For my last few projects, I was forced to work without automated testing and honestly, it was embarrassing to have the client email me after a code push to say that the application was breaking in places where I hadn’t even touched the code.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, in cases where my my client either had no budget or intention of adding any automated test framework, I started testing the whole website’s basic functionality by sending an HTTP request to each individual page, parsing the response headers and looking for the ‘200’ response. It sounds plain and simple, but there is a lot you can do to ensure fidelity without actually having to write any tests, unit, functional, or integration.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="While there are merits to all three types of testing, they don’t get written in most of projects." src="https://assets.toptal.io/uploads/blog/image/92479/toptal-blog-image-1462501532861-d8d9fb3007704d1b4809d568577ab9aa.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /><span style="background-color: #fafafa; color: #505050; font-size: 1.3em; line-height: 1.5em; text-align: center;">Tests don’t get written in a lot of contract projects for a variety of reasons, so what can you do?</span></div>
<h2 id="automated-testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Automated Testing</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In web development, automated tests comprise of three major test types: unit tests, functional tests and integration tests. We often combine unit tests with functional and integration tests to make sure everything runs smoothly as a whole application. When these tests are run in unison, or sequentially (preferably with single command or click), we start calling them automated tests, unit or not.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Largely the purpose of these tests (at least in web dev) is to make sure all application pages are rendered without trouble, free from fatal (application halting) errors or bugs.</div>
<h3 id="unit-testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Unit Testing</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Unit testing is a software development process in which the smallest parts of code – units – are independently tested for correct operation. Here’s an example in Ruby:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 835px;">test “should <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> active users” <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
active_user = create(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:user</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">active:</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>)
non_active_user = create(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:user</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">active:</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>)
result = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">User</span>.active
assert_equal result, [active_user]
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<h3 id="functional-testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Functional Testing</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Functional testing is a technique used to check the features and functionality of the system or software, designed to cover all user interaction scenarios, including failure paths and boundary cases.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Note: all our examples are in Ruby.</span></div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 835px;">test <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"should get index"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
get <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:index</span>
assert_response <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:success</span>
assert_not_nil assigns(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:object</span>)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<h3 id="integration-testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Integration Testing</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Once the modules are unit tested, they are integrated one by one, sequentially, to check the combinational behavior, and to validate that the requirements are implemented correctly.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 835px;">test <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"login and browse site"</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">do</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># login via https</span>
https!
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/login"</span>
assert_response <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:success</span>
post_via_redirect <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/login"</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">username:</span> users(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:david</span>).username, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">password:</span> users(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:david</span>).password
assert_equal <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/welcome'</span>, path
assert_equal <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Welcome david!'</span>, flash[<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:notice</span>]
https!(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>)
get <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"/articles/all"</span>
assert_response <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:success</span>
assert assigns(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:articles</span>)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<h2 id="tests-in-an-ideal-world" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Tests in an Ideal World</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Testing is widely accepted in the industry and it makes sense; good tests let you:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Quality assure your whole application with the least human effort</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Identify bugs more easily because you know exactly where your code is breaking from test failures</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Create automatic documentation for your code</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Avoid <a href="http://stackoverflow.com/questions/67299/is-unit-testing-worth-the-effort" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">‘coding constipation’</a>, which, according to some dude on Stack Overflow, is a humorous way of saying, “when you don’t know what to write next, or you have a daunting task in front of you, start by writing small.”</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I could go on and on about how awesome tests are, and how they changed the world and yada yada yada, but you get the point. Conceptually, tests are awesome.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #3863a0; font-size: 1.3em; line-height: 1.3em; text-align: center;">Tests in the Real World</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
While there are merits to all three types of testing, they don’t get written in most of projects. Why? Well, let me break it down:</div>
<h3 id="timedeadlines" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Time/Deadlines</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Everyone has deadlines, and writing fresh tests can get in the way of meeting one. It can take time and a half (or more) to write an application <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">and</em> its respective tests. Now, some of you do not agree with this, citing time saved ultimately, but I don’t think this is the case and I’ll explain why in ‘Difference of Opinion’.</div>
<h3 id="client-issues" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Client Issues</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Often, the client doesn’t really understand what testing is, or why it has value for the application. Clients tend to be more concerned with rapid product delivery and therefore see programmatic testing as counterproductive.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Or, it may be as simple as the client not having the budget to pay for the extra time needed to implement these tests.</div>
<h3 id="lack-of-knowledge" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Lack of Knowledge</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There is a sizeable tribe of developers in the real world that doesn’t know testing exists. At every conference, meetup, concert, (even in my dreams), I meet developers that don’t know how to write tests, don’t know what to test, don’t know how to setup the framework for testing, and so on. Testing isn’t exactly taught in schools, and it can be a hassle to set up/learn the framework to get them running. So yes, there’s a definite barrier to entry.</div>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-label is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 835px;">
<div class="embeddable_form-label_title" style="border: 0px; box-sizing: border-box; color: #2557a1; display: inline-block; font-size: 17px; font-weight: 700; line-height: 23px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px 0px 10px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 835px;">
<div class="embeddable_form-label is-header" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div class="embeddable_form-label" style="border: 0px; box-sizing: border-box; color: #313131; display: inline-block; font-size: 17px; font-style: italic; line-height: 19px; margin: 0px 10px 0px 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a data-role="preferences_link" href="https://www.toptal.com/freelance/http-request-testing-a-survival-tool#" style="border: 0px; box-sizing: border-box; color: #3976cb; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;"></a></div>
</div>
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 835px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item is-counter" style="background-color: #3863a0; border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-radius: 4px 0px 0px 4px; border-right-color: rgb(56, 99, 160) !important; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; color: white; flex-shrink: 0; height: 50px; line-height: 15px; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 10px 0px; position: relative; text-align: center; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;" title="Total number of shares"><span class="social_share-item_num" data-role="counter_num" style="border: 0px; box-sizing: border-box; display: block; font-size: 14px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: opacity 0.3s; vertical-align: baseline;"></span><span class="social_share-item_text" data-role="counter_text" style="border: 0px; box-sizing: border-box; display: block; font-size: 9px; margin: 0px; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; text-transform: uppercase; transition: opacity 0.3s; vertical-align: baseline;"></span></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/twitter_83c6d4.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="its-a-lot-of-work" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
‘It’s a Lot of Work’</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Writing tests can be overwhelming for both new and experienced programmers, even for those world-changer genius types, and to top it off, writing tests isn’t exciting. One may think, “Why should I engage in unexciting busywork when I could be implementing a major feature with results that will impress my client?” It’s a tough argument.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Last, but not least, <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">it is hard to write tests</span> and computer-science students are not trained for it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Oh, and refactoring with unit tests is no fun.</div>
<h3 id="difference-in-opinion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Difference in Opinion</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In my opinion, unit testing makes sense for algorithmic logic but not so much for coordinating living code.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
People claim that even though you’re investing extra time up front in writing tests, it saves you hours later when debugging or changing code. I beg to differ and offer one question: Is your code static, or ever changing?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For most of us, it’s ever changing. If you are writing successful software, you’re always adding features, changing existing ones, removing them, eating them, whatever, and to accommodate these changes, you must keep changing your tests, and changing your tests takes time.</div>
<h2 id="but-you-need-some-kind-of-testing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But, You Need Some Kind Of Testing</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
No one will argue that lacking any sort of testing is the worst possible case. After making changes in your code, you need to confirm that it actually works. A lot of programmers try to manually test the basics: Is the page rendering in the browser? Is the form being submitted? Is the correct content being displayed? And so on, but in my opinion, this is barbaric, inefficient and labour intensive.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="No one will argue that lacking any sort of testing is the worst possible case." src="https://assets.toptal.io/uploads/blog/image/92478/toptal-blog-image-1462501444795-32e8ef540fcc3a10ae836f703865dc11.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h2 id="what-i-use-instead" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What I Use Instead</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The purpose of testing a web app, be it manually or automated, is to confirm that any given page is rendered in the user’s browser without any fatal errors, and that it shows its content correctly. One way (and in most cases, an easier way) to achieve this is by sending HTTP requests to the endpoints of the app and parse the response. The response code tells you whether the page was delivered successfully. It’s easy to test for content by parsing the response body of the HTTP request and searching for specific text string matches, or, you can be one step fancier and use web scraping libraries such as <a href="http://www.nokogiri.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">nokogiri</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If some endpoints require a user login, you can use libraries designed for automating interactions (ideal when doing integration tests) such as <a href="http://wwwsearch.sourceforge.net/mechanize/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">mechanize</a> to login or click on certain links. Really, in the big picture of automated testing, this looks a lot like integration or functional testing (depending on how you use them), but it’s a lot quicker to write and can be included in an existing project, or added to a new one, with less effort than setting up whole testing framework. Spot on!</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: 1.2em; line-height: 1.5em; text-align: center;">Edge cases present another problem when dealing with large databases with a wide range of values; testing whether our application is working smoothly across all anticipated datasets can be daunting.</span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One way to go about it is to anticipate all the edge cases (which is not merely difficult, it’s often impossible) and write a test for each one. This could easily become hundreds of lines of code (imagine the horror) and cumbersome to maintain. Yet, with HTTP requests and just one line of code, you can test such edge cases directly on the data from production, downloaded locally on your development machine or on a staging server.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now of course, this testing technique is not a silver bullet and has lots of shortcomings, the same as any other method, but I find these types of tests faster, and easier, to write and modify.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="With HTTP requests & one line of code, you can test such edge cases directly on data from production." src="https://assets.toptal.io/uploads/blog/image/92474/toptal-blog-image-1462499898952-ef63dc98acf2d60af98ce5fc110e94a7.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<h3 id="in-practice-testing-with-http-requests" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In Practice: Testing with HTTP requests</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since we’ve already established that writing code without any kind of accompanying tests isn’t a good idea, my very basic go-to test for an entire application is to send HTTP requests to all its pages locally and parse the response headers for a <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">200</code> (or desired) code.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For example, if we were to write the above tests (the ones looking for specific content and a fatal error) with an HTTP request instead (in Ruby), it would be something like this:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 835px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># testing for fatal error</span>
http_code = `curl -<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">X</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{route[:method]} -s -o /dev/null -w "%{http_code}" #{Rails.application.routes.url_helpers.articles_url(host: 'localhost', port: 3000)</span>
}`
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> http_code !~ <span class="hljs-regexp" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/200/</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> “articles_url returned with <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{http_code} http code.”</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># testing for content</span>
active_user = create(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:user</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name:</span> “user1”, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">active:</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>)
non_active_user = create(<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:user</span>, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">name:</span> “user2”, <span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">active:</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>)
content = `curl <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{Rails.application.routes.url_helpers.active_user_url(host: 'localhost', port: 3000)</span>
}`
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> content !~ <span class="hljs-regexp" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{active_user.name}</span>/</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> “<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Content</span> mismatch active user <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{active_user.name} not found in text body” #You can customise message to your liking</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> content =~ <span class="hljs-regexp" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">/<span class="hljs-subst" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{non_active_user.name}</span>/</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> “<span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Content</span> mismatch non active user <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{active_user.name} found in text body” #You can customise message to your liking</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">end</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The line <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">curl -X #{route[:method]} -s -o /dev/null -w "%{http_code}" #{Rails.application.routes.url_helpers.articles_url(host: 'localhost', port: 3000) }</code>covers a lot of test cases; any method raising an error on the article’s page will be caught here, so it effectively covers hundreds of lines of code in one test.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The second part, which catches the content error specifically, can be used multiple times to check the content on a page. (More complex requests can be handled using <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">mechanize</code>, but that’s beyond the scope of this blog.)</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, in cases where you want to test if a specific page works on a large, varied set of database values (for example, your article page template is working for all the articles in the production database), you could do:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-ruby hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 835px;">ids = <span class="hljs-constant" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Article</span>.all.select { |post| `curl -s -o /dev/null -w “<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">%{http_code}</span>” <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">#{Rails.application.routes.url_helpers.article_url(post, host: 'localhost', port: 3000)</span>
}`.to_i != <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">200</span>).map(&<span class="hljs-symbol" style="border: 0px; box-sizing: border-box; color: #cb4b16; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:id</span>)
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> ids
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This will return an array of IDs of all the articles in the database that were not rendered, so now you can manually go to the specific article page and check out the problem.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, I understand that this way of testing might not work in certain cases, such as testing a standalone script or sending an email, and it is undeniably slower than unit tests because we are making direct calls to an endpoint for each test, but when you can’t have unit tests, or functional tests, or both, this is better than nothing.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How would you go about structuring these tests? With small, non-complex projects, you can write all your tests in one file and run that file each time before you commit your changes, but most projects will require a suite of tests.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I usually write two to three tests per endpoint, depending on what I’m testing. You can also try testing individual content (similar to unit testing), but I think that would be redundant and slow since you will be making an HTTP call for every unit. But, on the other hand, they will be cleaner and easy to understand.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I recommend putting your tests in your regular test folder with each major end point having its own file (in Rails, for example, each model/controller would have one file each), and this file can be divided into three parts according to what we are testing. I often have at least three tests:</div>
<h4 id="test-one" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Test One</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Check that the page returns without any fatal errors.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Test one checks that the page returns without any fatal errors." src="https://assets.toptal.io/uploads/blog/image/92477/toptal-blog-image-1462500443914-17e7edad0f75be6f60ceca51a0bce63f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note how I made a list of all the endpoints for <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Post</code> and iterated over it to check that each page is rendered without any error. Assuming everything went well, and all the pages were rendered, you will see something like this in the terminal: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of failed url(s) -- []</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If any page is not rendered, you will see something like this (in this example, the <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">posts/index page</code> has error and hence is not rendered): <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of failed url(s) -- [{:url=>”posts_url”, :params=>[], :method=>”GET”, :http_code=>”500”}]</code></div>
<h4 id="test-two" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Test Two</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Confirm that all the expected content is there:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Test two confirms that all the expected content is there." src="https://assets.toptal.io/uploads/blog/image/92476/toptal-blog-image-1462500340971-26349a96459a1534a52f0dddb3e967c6.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If all the content we expect is found on the page, the result looks like this (in this example we make sure <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">posts/:id</code> has a post title, description and a status): <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of content(s) not found on Post#show page with post id: 1 -- []</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If any expected content is not found on the page (here we expect the page to show status of post - ‘Active’ if post is active, ‘Disabled’ if post is disabled) the result looks like this: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of content(s) not found on Post#show page with post id: 1 -- [“Active”]</code></div>
<h4 id="test-three" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Test Three</h4>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Check that the page renders across all datasets (if any):</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Test 3 checks that the page renders across all datasets." src="https://assets.toptal.io/uploads/blog/image/92475/toptal-blog-image-1462500024549-7fa9ce001b0d2acfebe8e761d78f562e.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If all the pages are rendered without any error, we will get an empty list: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of post(s) with error in rendering -- []</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If the content of some of the records has a problem rendering (in this example, pages with the ID 2 and 5 are giving an error) the result looks like this: <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">➜ sample_app git:(master) ✗ ruby test/http_request/post_test.rb List of post(s) with error on rendering -- [2,5]</code></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you want to fiddle around with the above demonstration code, <a href="https://github.com/rubydog/sample_app" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">here’s my github project</a>.</div>
<h3 id="so-which-is-better-it-depends" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So Which Is Better? It Depends…</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">HTTP Request testing might be your best bet if:</span></div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You’re working with a web app</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You’re in a time crunch and want to write something fast</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You’re working with a big project, pre-existing project where tests were not written, but you still want some way to check code</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Your code involves simple request and response</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You don’t want to spend a large portion of your time maintaining tests (I read somewhere unit test = maintenance hell, and I partially agree with him/her)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You want to test if an application works across all the values in an existing database</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Traditional testing is ideal when:</span></div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You’re dealing with something other than a web application, such as scripts</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You’re writing complex, algorithmic code</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">You have time and budget to dedicate to writing tests</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">The business requires bug-free or a low-error rate (finance, large user base)</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Thanks for reading the article; you should now have a method for testing you can default to, one you can count on when you’re pressed for time.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #303030;">This article was written by </span><span style="border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border: 0px; box-sizing: border-box; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/resume/bhushan-lodha" style="border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border: 0px; box-sizing: border-box; color: #3863a0; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition-property: color, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">BHUSHAN LODHA</a><span style="color: #929292;">, </span></span><span style="color: #303030; font-size: 19.2px; line-height: 28.8px;">a <a href="https://www.toptal.com/freelance/http-request-testing-a-survival-tool" target="_blank">Toptal</a><a href="https://www.toptal.com/ruby" target="_blank"> Ruby developer</a>.</span></div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-5412505838746501492016-05-18T14:05:00.000+07:002016-05-18T14:05:28.647+07:00Introduction To Concurrent Programming: A Beginner's Guide<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What is concurrent programing? Simply described, it’s when you are doing more than one thing at the same time. Not to be confused with parallelism, concurrency is when multiple sequences of operations are run in overlapping periods of time. In the realm of programming, concurrency is a pretty complex subject. Dealing with constructs such as threads and locks and avoiding issues like race conditions and deadlocks can be quite cumbersome, making concurrent programs difficult to write. Through concurrency, programs can be designed as independent processes working together in a specific composition. Such a structure may or may not be made parallel; however, achieving such a structure in your program offers numerous advantages.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Introduction To Concurrent Programming" src="https://assets.toptal.io/uploads/blog/image/92452/toptal-blog-image-1461932189484-7656f0dd3f03c9ffa5bfed53580561a7.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this article, we will take a look at a number of different models of concurrency, how to achieve them in various programming languages <a href="https://en.wikipedia.org/wiki/List_of_concurrent_and_parallel_programming_languages" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">designed for concurrency</a>.</div>
<h2 id="shared-mutable-state-model" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Shared Mutable State Model</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s look at a simple example with a counter and two threads that increase it. The program shouldn’t be too complicated. We have an object that contains a counter that increases with method increase, and retrieves it with method get and two threads that increase it.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Counting.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Counting</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
class Counter {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> counter = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">increment</span>() { counter++; }
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span>() { <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> counter; }
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Counter counter = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Counter();
class CountingThread extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> x = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; x < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">500000</span>; x++) {
counter.increment();
}
}
}
CountingThread t1 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CountingThread();
CountingThread t2 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CountingThread();
t1.start(); t2.start();
t1.join(); t2.join();
System.out.println(counter.get());
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This naive program is not as naive as it seems at first glance. When I run this program more times I get different results. There are three values after three executions on my laptop.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">java Counting
553706
java Counting
547818
java Counting
613014
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What is the reason for this <a href="http://blog.regehr.org/archives/490" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">unpredictable behavior</a>? The program increases the counter in one place, in method increase that uses command counter++. If we look at the command byte code we would see that it consists of several parts:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Read counter value from memory</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Increase value locally</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Store counter value in memory</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92453/toptal-blog-image-1461932282027-b5a1013d82a53285f068d9e5e276068d.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now we can imagine what can go wrong in this sequence. If we have two threads that independently increase the counter then we could have this scenario:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Counter value is 115</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread reads the value of the counter from the memory (115)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread increases the local counter value (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread reads the value of the counter from the memory (115)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread increases the local counter value (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread saves the local counter value to the memory (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread saves the local counter value to the memory (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Value of the counter is 116</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this scenario, two threads are intertwined so that the counter value is increased by 1, but the counter value should be increased by 2 because each thread increases it by 1. Different threads intertwining influences the result of the program. The reason of the program’s unpredictability is that the program has no control of the thread intertwining but operating system. Every time the program is executed, threads can intertwine differently. In this way we introduced accidental unpredictability (non-determinism) to the program.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To fix this accidental unpredictability (non-determinism), the program must have control of the thread intertwining. When one thread is in the method increase another thread must not be in the same method until the first comes out of it. In that way we serialize access to the method increase.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// CountingFixed.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">CountingFixed</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
class Counter {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> counter = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">increase</span>() { counter++; }
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span>() { <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> counter; }
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Counter counter = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Counter();
class CountingThread extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">500000</span>; i++) {
counter.increment();
}
}
}
CountingThread thread1 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CountingThread();
CountingThread thread2 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CountingThread();
thread1.start(); thread2.start();
thread1.join(); thread2.join();
System.out.println(counter.get());
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another solution is to use a counter which can increase atomically, meaning operation can not be separated into multiple operations. In this way, we don’t need to have blocks of code that need to synchronize. Java has atomic data types in java.util.concurrent.atomic namespace, and we’ll use AtomicInteger.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// CountingBetter.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> java.util.concurrent.atomic.AtomicInteger;
class CountingBetter {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> AtomicInteger counter = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> AtomicInteger(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>);
class CountingThread extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> viod <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">500000</span>; i++) {
counter.incrementAndGet();
}
}
}
CountingThread thread1 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CountingThread();
CountingThread thread2 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> CoutningThread();
thread1.start(); thread2.start();
thread1.join(); thread2.join();
System.out.println(counter.get());
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Atomic integer has the operations that we need, so we can use it instead of the Counter class. It is interesting to note that all methods of atomicinteger do not use locking, so that there is no possibility of deadlocks, which facilitates the design of the program.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Using <a href="https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">synchronized</a> keywords to synchronize critical methods should resolve all problems, right? Let’s imagine that we have two accounts that can deposit, withdraw and transfer to another account. What happens if at the same time we want to transfer money from one account to another and vice versa? Let’s look at an example.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// Deadlock.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deadlock</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
class Account {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> balance = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Account</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> balance) { <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>.balance = balance; }
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">deposit</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) { balance += amount; }
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">withdraw</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (balance >= amount) {
balance -= amount;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span>(Account destination, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (balance >= amount) {
balance -= amount;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span>(destination) {
destination.balance += amount;
};
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getBalance</span>() { <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> balance; }
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Account bob = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Account(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">200000</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Account joe = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Account(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">300000</span>);
class FirstTransfer extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>; i++) {
bob.transfer(joe, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>);
}
}
}
class SecondTransfer extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>; i++) {
joe.transfer(bob, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>);
}
}
}
FirstTransfer thread1 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> FirstTransfer();
SecondTransfer thread2 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> SecondTransfer();
thread1.start(); thread2.start();
thread1.join(); thread2.join();
System.out.println(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Bob's balance: "</span> + bob.getBalance());
System.out.println(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Joe's balance: "</span> + joe.getBalance());
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this program on my laptop it usually gets stuck. Why does this happen? If we look closely, we can see that when we transfer money we are entering into the transfer method that is synchronized and locks access to all synchronized methods on the source account, and then locks destination account which locks access to all synchronized methods on it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92454/toptal-blog-image-1461932308078-e825242ab0e631cecddedcf9fb360192.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Imagine the following scenario:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread calls transfer on Bob’s account to Joe’s account</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread calls transfer on Joe’s account to Bob’s account</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread decreases amount from Joe’s account</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Second thread goes to deposit amount to Bob’s account but waits for first thread to complete transfer.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread decreases amount from Bob’s account</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">First thread goes to deposit amount to Joe’s account but waits for second thread to complete transfer.</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this scenario, one thread is waiting for another thread to finish transfer and vice versa. They are stuck with each other and the program cannot continue. This is called <a href="https://docs.oracle.com/javase/tutorial/essential/concurrency/deadlock.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">deadlock</a>. To avoid deadlock it is necessary to lock accounts in the same order. To fix the program we’ll give each account a unique number so that we can lock accounts in the same order when transferring the money.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// DeadlockFixed.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> java.util.concurrent.atomic.AtomicInteger;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">DeadlockFixed</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> AtomicInteger counter = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> AtomicInteger(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>);
class Account {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> balance = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> order;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Account</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> balance) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>.balance = balance;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>.order = counter.getAndIncrement();
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">deposit</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) { balance += amount; }
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">withdraw</span>(<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (balance >= amount) {
balance -= amount;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">boolean</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span>(Account destination, <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> amount) {
Account first;
Account second;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>.order < destination.order) {
first = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>;
second = destination;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">else</span> {
first = destination;
second = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">this</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span>(first) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span>(second) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (balance >= amount) {
balance -= amount;
destination.balance += amount;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">true</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">false</span>;
}
}
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">synchronized</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getBalance</span>() { <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> balance; }
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Account bob = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Account(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">200000</span>);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Account joe = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Account(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">300000</span>);
class FirstTransfer extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>; i++) {
bob.transfer(joe, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>);
}
}
}
class SecondTransfer extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> (<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">int</span> i = <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">0</span>; i < <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">100000</span>; i++) {
joe.transfer(bob, <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>);
}
}
}
FirstTransfer thread1 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> FirstTransfer();
SecondTransfer thread2 = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> SecondTransfer();
thread1.start(); thread2.start();
thread1.join(); thread2.join();
System.out.println(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Bob's balance: "</span> + bob.getBalance());
System.out.println(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Joe's balance: "</span> + joe.getBalance());
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Due to the unpredictability of such mistakes, they sometimes happen, but not always and they are difficult to reproduce. If the program behaves unpredictably, it is usually caused by concurrency which introduces accidental non-determinism. To avoid accidental non-determinism we should in advance design program to take into account all intertwinings.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
An example of a program that has an accidental non-determinism.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-java hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// NonDeteminism.java</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">//</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-class" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">class</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">NonDeterminism</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">static</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">main</span>(String[] args) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">throws</span> InterruptedException {
class Container {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> String value = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Empty"</span>;
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">final</span> Container container = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> Container();
class FastThread extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
container.value = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Fast"</span>;
}
}
class SlowThread extends Thread {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">public</span> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">void</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">run</span>() {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">try</span> {
Thread.sleep(<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">50</span>);
}
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">catch</span>(Exception e) {}
container.value = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Slow"</span>;
}
}
FastThread fast = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> FastThread();
SlowThread slow = <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">new</span> SlowThread();
fast.start(); slow.start();
fast.join(); slow.join();
System.out.println(container.value);
}
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This program has accidental non-determinism in it. The last value entered in the container will be displayed.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">java NonDeterminism
Slow
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Slower threads will enter the value later, and this value will be printed (Slow). But this needs not be the case. What if the computer simultaneously executes another program that needs a lot of CPU resources? We have no guarantee that it will be the slower thread that enters value last because it is controlled by operating system, not the program. We can have situations where the program works on one computer and on the other behaves differently. Such errors are difficult to find and they cause headaches for developers. For all these reasons this concurrency model is very difficult to do right.</div>
<h2 id="functional-way" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Functional Way</h2>
<h3 id="parallelism" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Parallelism</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s look at another model that functional languages are using. For example we will use <a href="https://clojure.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Clojure</a>, that can be interpreted using the tool <a href="http://leiningen.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Leiningen</a>. Clojure is a very interesting language with good support for concurrency. The previous concurrency model was with shared mutable state. Classes that we use can also have a hidden state that mutates that we don’t know about, because it is not evident from their API. As we have seen, this model can cause accidental non-determinism and deadlocks if we are not careful. Functional languages have data types that don’t mutate so it can be safely shared without the risk that they will change. Functions have properties as well as other data types. Functions can be created during program execution and passed as parameter to another function or return as a result of the function call.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92455/toptal-blog-image-1461932456572-18f7e49ad8244d1836a4ba2b6980db19.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Basic primitives for concurrent programing are future and promise. Future executes a block of code in another thread and returns an object for the future value that will be entered when the block gets executed.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; future.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">let</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[a <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Started A"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Thread/sleep</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1000</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Finished A"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span>)</span>
b <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Started B"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Thread/sleep</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2000</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Finished B"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 3</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 4</span>)</span>)</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Waiting for futures"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> @a @b)</span>)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I execute this script the output is:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Started A
Started B
Waiting for futures
Finished A
Finished B
10
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example we have two future blocks that are executed independently. Program only blocks when reading the value from the future object that is not yet available. In our case, awaiting both results of future blocks to be summed. Behavior is predictable (deterministic) and will always give the same result because there is no shared mutable state.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another primitive that is used for concurrency is a promise. Promise is a container in which one can put a value once. When reading promises, the thread will wait until the value of the promise gets filled.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; promise.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> result <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">promise</span></span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The result is: "</span> @result)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Thread/sleep</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2000</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">deliver</span> result<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 42</span>)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example, the <a href="https://clojuredocs.org/clojure.core/future" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">future</a> will wait to print the result as long as the promise not to be saved value. After two seconds, in the promise will be stored value 42 to be printed in the future thread. Using <a href="https://clojuredocs.org/clojure.core/promise" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">promises</a>can lead to deadlock as opposed to the future, so be careful when using promise.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; promise-deadlock.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> promise-result <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">promise</span></span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> future-result
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The result is: "</span> + @promise-result)</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
13</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Future result is: "</span> @future-result)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">deliver</span> result<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 42</span>)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example, we are using the result of the future and the result of the promise. The order of setting and reading values is that the main thread is waiting for a value from the future thread and future thread is waiting for a value from the main thread. This behavior will be predictable (deterministic) and will be played each time the program executes which makes it easier to find and remove error.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Using the future allows the program to continue with the exercise until it needs the result of the execution of the future. This results in faster program execution. If you have multiple processors with the future, you can make parallel execution of program that have predictable (deterministic) behavior (each time gives the same result). That way we better exploit the power of the computer.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; fibonacci.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> fibonacci<span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[a]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><=</span></span> a<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
1</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">-</span></span> a<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1</span>)</span>)</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">-</span></span> a<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span>)</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Start serial calculation"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">time</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The result is: "</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 36</span>)</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 36</span>)</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Start parallel calculation"</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> parallel-fibonacci<span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> result-1 <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 36</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> result-2 <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fibonacci</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 36</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> @result-1 @result-2)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">time</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The result is: "</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">parallel-fibonacci</span>)</span>)</span>)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example you can see how the use of future can make better use of a computer’s speed. We have two Fibonacci numbers that add up. We can see that program calculates the result twice, the first time sequentially in a single thread, and the second time in parallel in two threads. As my laptop has a multicore processor, parallel execution works twice as fast as sequential calculation.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The result of executing this script on my laptop:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Start serial calculation
The result is: 29860704
"Elapsed time: 2568.816524 msecs"
Start parallel calculation
The result is: 29860704
"Elapsed time: 1216.991448 msecs"</code></pre>
<div class="embeddable_form-wrapper for-post" data-role="blog_subscribe" data-view="blog_subscribe#form" style="background-color: white; border-radius: 6px; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; line-height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-step is-confirmation" data-place="@blog_subscribe" data-role="confirmation" style="border: 0px; box-sizing: border-box; height: 0px; margin: 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 0px; vertical-align: baseline;">
<div class="embeddable_form-row is-success" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline; width: 828px;">
<ul class="social_share is-horizontal is-loaded" data-rss-url="https://www.toptal.com/blog.rss" data-twitter-username="@toptalllc" data-url="https://www.toptal.com/blog" data-view="layout#social_share" data-youtube-channel-url="https://www.youtube.com/channel/UCNqm_euTHZz3o5OnKhUS-oA" style="-webkit-box-direction: normal; -webkit-box-orient: horizontal; -webkit-box-pack: start; border: 0px; box-sizing: border-box; display: flex; flex-direction: row; font-size: 1.2em; justify-content: flex-start; list-style: none; margin: 0px; max-width: 300px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="facebook" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Facebook"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/facebook_dc66c9.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-bottom-color: rgb(236, 236, 236); border-bottom-style: solid; border-left-color: rgb(236, 236, 236); border-left-style: solid; border-top-color: rgb(236, 236, 236); border-top-style: solid; border-width: 1px 0px 1px 1px; box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0.75em; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="google_plus" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Share on Google Plus"><img class="social_share-item_image" height="50" src="https://assets.toptal.io/assets/front/static/public/primitives/social/share_bar/google_plus_355fb0.png" style="border: 0px; box-sizing: border-box; display: block; height: 50px; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline; width: 50px;" width="50" /></a></li>
<li class="social_share-item" style="border-radius: 0px 4px 4px 0px; border: 1px solid rgb(236, 236, 236); box-sizing: border-box; flex-shrink: 0; height: 50px; line-height: 1.5em; list-style-type: disc; margin-bottom: 0px; margin-left: 30px; margin-right: 0px !important; margin-top: 0px !important; min-height: 0px; min-width: 0px; padding: 0px; text-shadow: rgba(0, 0, 0, 0.498039) 0px 1px 2px; transition: box-shadow 0.2s; vertical-align: baseline; width: 50px;"><a class="social_share-item_link" data-role="link" data-type="twitter_follow" href="https://www.blogger.com/null" style="border: 0px; box-sizing: border-box; color: #3976cb; cursor: pointer; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" title="Follow Toptal on Twitter"></a></li>
</ul>
</div>
</div>
</form>
</div>
<h3 id="concurrency" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Concurrency</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To support concurrency and unpredictability in the Clojure programming language, we must use a data type that is variable so other threads can see the changes. The simplest variable data type is atom. <a href="http://clojure.org/reference/atoms" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Atom</a>is a container which always has the value that can be replaced by another value. The value can be replaced by entering a new value or by calling a function that takes the old value and returns new value which is more frequently used. It is interesting that atom is implemented without locking and it is safe to use in threads, which means that it is impossible to reach deadlock. Internally, atom uses java.util.concurrent.AtomicReference library. Let’s look at a counter example implemented with atom.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; atom-counter.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> counter <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> attempts <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> counter-increases<span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 500000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> counter <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fn</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[counter]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> attempts inc)</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; side effect DO NOT DO THIS</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">inc</span></span> counter)</span>)</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> first-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">counter-increases</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> second-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">counter-increases</span>)</span>)</span>)</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; Wait for futures to complete</span>
@first-future
@second-future
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; Print value of the counter</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The counter is: "</span> @counter)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Number of attempts: "</span> @attempts)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The result of the script execution on my laptop:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">The counter is: 1000000
Number of attempts: 1680212
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example we use an atom that contains the value of the counter. The counter increases with (swap! counter inc). Swap function works like this: 1. take the counter value and preserve it 2. for this value calls given function that calculates the new value 3. to save new value, it uses atomic operation that checks whether the old value has changed 3a. if the value has not changed it enters a new value 3b. if the value is changed in the meantime, then go to step 1 We see that the function can be called again if the value is changed in the meantime. The value can only be changed from another thread. Therefore, it is essential that the function which calculates a new value has no side effects so that it does not matter if it gets called more times. One limitation of atom is that it synchronizes changes to one value.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; atom-acocunts.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> bob <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 200000</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> joe <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 300000</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> inconsistencies <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[source destination amount]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">not=</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> @bob @joe)</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 500000</span>)</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> inconsistencies inc)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> source - amount)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> destination + amount)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> first-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> bob joe<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> second-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> joe bob<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> first-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">first-transfer</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> second-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">second-transfer</span>)</span>)</span>)</span>
@first-future
@second-future
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Bob has in account: "</span> @bob)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Joe has in account: "</span> @joe)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Inconsistencies while transfer: "</span> @inconsistencies)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I execute this script I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Bob has in account: 100000
Joe has in account: 400000
Inconsistencies while transfer: 36525
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this example we can see how we change more atoms. At one point, inconsistency can happen. The sum of two accounts at some time is not the same. If we have to coordinate changes of multiple values there are two solutions:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Place more values in one atom</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Use references and software transactional memory, as we shall see later</li>
</ol>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; atom-accounts-fixed.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> accounts <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">{<span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:bob</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 200000</span>, <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:joe</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 300000</span>}</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> inconsistencies <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[source destination amount]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">let</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[deref-accounts @accounts]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">not=</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span></span> deref-accounts <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:bob</span>)</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span></span> deref-accounts <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:joe</span>)</span>)</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 500000</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> inconsistencies inc)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> accounts
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fn</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[accs]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">update</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">update</span> accs source - amount)</span> destination + amount)</span>)</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> first-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:bob</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:joe</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> second-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:joe</span> <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:bob</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> first-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">first-transfer</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> second-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">second-transfer</span>)</span>)</span>)</span>
@first-future
@second-future
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Bob has in account: "</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span></span> @accounts <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:bob</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Joe has in account: "</span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">get</span></span> @accounts <span class="hljs-attribute" style="border: 0px; box-sizing: border-box; color: #b58900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">:joe</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Inconsistencies while transfer: "</span> @inconsistencies)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script on my computer I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Bob has in account: 100000
Joe has in account: 400000
Inconsistencies while transfer: 0
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the example, coordination has been resolved so that we put more value using a map. When we transfer money from the account, we change all acounts at the time so that it will never happen that the sum of money is not the same.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The next variable data type is agent. Agent behaves like an atom only in that the function that changes the value is executed in a different thread, so that it takes some time for change to become visible. Therefore, when reading the value of the agent it is necessary to call a function that waits until all functions that change the value of the agent are executed. Unlike atoms function that changes the value is called only once and therefore can have side effects. This type can also synchronize one value and cannot deadlock.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; agent-counter.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> counter <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">agent</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> attempts <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> counter-increases<span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 500000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">send</span></span> counter <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fn</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[counter]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> attempts inc)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">inc</span></span> counter)</span>)</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> first-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">counter-increases</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> second-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">counter-increases</span>)</span>)</span>)</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; wait for futures to complete</span>
@first-future
@second-future
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; wait for counter to be finished with updating</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">await</span></span> counter)</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; print the value of the counter</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"The counter is: "</span> @counter)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Number of attempts: "</span> @attempts)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script on my laptop I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">The counter is: 1000000
Number of attempts: 1000000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This example is the same as the implementation of the counter with the atom. Only difference is that here we are waiting for all agent changes to complete before reading the final value using await.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The last variable data type are references. Unlike atoms, references can synchronize changes to multiple values. Each operation on reference should be in a transaction using dosync. This way of changing data is called software transactional memory or abbreviated STM. Let’s look at an example with the money transfer in the accounts.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-clojure hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; stm-accounts.clj</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">;</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> bob <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ref</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 200000</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> joe <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">ref</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 300000</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> inconsistencies <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> attempts <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">atom</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> transfers <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">agent</span></span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 0</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[source destination amount]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dosync</span></span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> attempts inc)</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; side effect DO NOT DO THIS</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">send</span></span> transfers inc)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">when</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">not=</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">+</span></span> @bob @joe)</span><span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 500000</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">swap!</span></span> inconsistencies inc)</span>)</span> <span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">; side effect DO NOT DO THIS</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">alter</span></span> source - amount)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">alter</span></span> destination + amount)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> first-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> bob joe<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 2</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">defn</span></span> second-transfer <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">dotimes</span></span> <span class="hljs-collection" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">[cnt<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 100000</span>]</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">transfer</span> joe bob<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"> 1</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> first-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">first-transfer</span>)</span>)</span>)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span></span> second-future <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">future</span></span> <span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">second-transfer</span>)</span>)</span>)</span>
@first-future
@second-future
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-built_in" style="border: 0px; box-sizing: border-box; color: #dc322f; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">await</span></span> transfers)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Bob has in account: "</span> @bob)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Joe has in account: "</span> @joe)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Inconsistencies while transfer: "</span> @inconsistencies)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Attempts: "</span> @attempts)</span>
<span class="hljs-list" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(<span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">println</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"Transfers: "</span> @transfers)</span>
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script, I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Bob has in account: 100000
Joe has in account: 400000
Inconsistencies while transfer: 0
Attempts: 330841
Transfers: 200000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Interestingly, there were more attempts than the number of transactions made. This is because the STM does not use locks, so if there is a conflict, (like two threads trying to change the same value) the transaction will be re-executed. For this reason, the transaction should not have side effects. We can see that the agent which value changes within the transaction behaves predictably. A function that changes the value of the agent will be evaluated as many times as there are transactions. The reason is that the agent is transaction aware. If transaction must have side effects, they should be put into function within the agent. In this way, the program will have predictable behavior. You probably think that you should always use STM, but experienced programmers will often use atoms because atoms are simpler and faster than STM. Of course, that’s if it is possible to make a program in that way. If you have side effects, then there’s no other choice than to use STM and agents.</div>
<h2 id="actor-model" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Actor Model</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The following model of concurrency is an <a href="https://en.wikipedia.org/wiki/Actor_model" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">actor model</a>. The principle of this model is similar to the real world. If we make a deal to create something with many people, for example a building, then each man at the construction site has their own role. A crowd of people is supervised by the supervisor. If a worker is injured at work, the supervisor will assign the job of the injured man to the others that are available. If necessary he may lead to the site a new man. On the site we have more people who do the work simultaneously (concurrently), but also talking to each other to synchronize. If we put work on the construction site into the program, then every person would be an actor who has a state and executes in its own process, and the talking would be replaced with messages. The popular programming language based on this model is Erlang. This interesting language has immutable data types and functions that have the same properties as other data types. Functions can be created during program execution and passed as arguments to another function or returned as result of function call. I will give examples in the<a href="https://www.toptal.com/elixir" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Elixir</a> language that uses the Erlang virtual machine, so I’ll have the same programming model as Erlang just different syntax. The three most important primitives in Elixir are spawn, send and receive. spawn executes function in the new process, send sends the message to the process and receive receives messages that are sent to the current process.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92456/toptal-blog-image-1461932487043-2b6331d3e76f6d7e9176bb11caf5ec1d.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first example with the actor model will be counter increased concurrently. To make a program with this model, it is necessary to make an actor have the value of the counter and receive message to set and retrieve the value of the counter, and have two actors who will simultaneously increase the value of the counter.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-elixir" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">#
# Counting.exs
#
defmodule Counting do
def counter(value) do
receive do
{:get, sender} ->
send sender, {:counter, value}
counter value
{:set, new_value} -> counter(new_value)
end
end
def counting(sender, counter, times) do
if times > 0 do
send counter, {:get, self}
receive do
{:counter, value} -> send counter, {:set, value + 1}
end
counting(sender, counter, times - 1)
else
send sender, {:done, self}
end
end
end
counter = spawn fn -> Counting.counter 0 end
IO.puts "Starting counting processes"
this = self
counting1 = spawn fn ->
IO.puts "Counting A started"
Counting.counting this, counter, 500_000
IO.puts "Counting A finished"
end
counting2 = spawn fn ->
IO.puts "Counting B started"
Counting.counting this, counter, 500_000
IO.puts "Counting B finished"
end
IO.puts "Waiting for counting to be done"
receive do
{:done, ^counting1} -> nil
end
receive do
{:done, ^counting2} -> nil
end
send counter, {:get, self}
receive do
{:counter, value} -> IO.puts "Counter is: #{value}"
end
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I execute this example I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Starting counting processes
Counting A started
Waiting for counting to be done
Counting B started
Counting A finished
Counting B finished
Counter is: 516827
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can see that in the end the counter is 516827 and not 1000000 as we expected. When I ran the script next time, I received 511010. The reason for this behavior is that the counter receives two messages: retrieve the current value and set the new value. To increase the counter, program needs to get the current value, increase it by 1 and set the increased value. Two processes read and write the value of the counter at the same time by using message that are sent to counter process. The order of messages that counter will receive is unpredictable, and the program cannot control it. We can imagine this scenario:</div>
<ol style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Counter value is 115</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process A reads the value of the counter (115)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process B reads the value of the counter (115)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process B increases the value locally (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process B sets increased value to the counter (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process A increases the value of the counter (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Process A sets increased value to the counter (116)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: decimal; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Counter value is 116</li>
</ol>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we look at the scenario, two processes increase the counter by 1, and counter gets increased in the end by 1 and not by 2. Such intertwinings can happen an unpredictable number of times and therefore the value of the counter is unpredictable. To prevent this behavior, the increase operation must be done by one message.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-elixir" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">#
# CountingFixed.exs
#
defmodule Counting do
def counter(value) do
receive do
:increase -> counter(value + 1)
{:get, sender} ->
send sender, {:counter, value}
counter value
end
end
def counting(sender, counter, times) do
if times > 0 do
send counter, :increase
counting(sender, counter, times - 1)
else
send sender, {:done, self}
end
end
end
counter = spawn fn -> Counting.counter 0 end
IO.puts "Starting counting processes"
this = self
counting1 = spawn fn ->
IO.puts "Counting A started"
Counting.counting this, counter, 500_000
IO.puts "Counting A finished"
end
counting2 = spawn fn ->
IO.puts "Counting B started"
Counting.counting this, counter, 500_000
IO.puts "Counting B finished"
end
IO.puts "Waiting for counting to be done"
receive do
{:done, ^counting1} -> nil
end
receive do
{:done, ^counting2} -> nil
end
send counter, {:get, self}
receive do
{:counter, value} -> IO.puts "Counter is: #{value}"
end
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By running this script I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Starting counting processes
Counting A started
Waiting for counting to be done
Counting B started
Counting A finished
Counting B finished
Counter is: 1000000
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can see that the counter has the correct value. The reason for predictable (deterministic) behavior is that the value of the counter increases by one message so that the sequence of messages to increase the counter will not affect its final value. Working with actor model, we have to pay attention to how messages can intertwine and careful design of messages and actions on messages to avoid accidental unpredictability (non-determinism).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How can we transfer money between two accounts with this model?</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-elixir" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">#
# Accounts.exs
#
defmodule Accounts do
def accounts(state) do
receive do
{:transfer, source, destination, amount} ->
accounts %{state | source => state[source] - amount , destination => state[destination] + amount}
{:amounts, accounts, sender } ->
send sender, {:amounts, for account <- accounts do
{account, state[account]}
end}
accounts(state)
end
end
def transfer(sender, accounts, source, destination, amount, times, inconsistencies) do
if times > 0 do
send accounts, {:amounts, [source, destination], self}
receive do
{:amounts, amounts} ->
if amounts[source] + amounts[destination] != 500_000 do
Agent.update(inconsistencies, fn value -> value + 1 end)
end
end
send accounts, {:transfer, source, destination, amount}
transfer(sender, accounts, source, destination, amount, times - 1, inconsistencies)
else
send sender, {:done, self}
end
end
end
accounts = spawn fn -> Accounts.accounts(%{bob: 200_000, joe: 300_000 }) end
{:ok, inconsistencies} = Agent.start(fn -> 0 end)
this = self
transfer1 = spawn fn ->
IO.puts "Transfer A started"
Accounts.transfer(this, accounts, :bob, :joe, 2, 100_000, inconsistencies)
IO.puts "Transfer A finished"
end
transfer2 = spawn fn ->
IO.puts "Transfer B started"
Accounts.transfer(this, accounts, :joe, :bob, 1, 100_000, inconsistencies)
IO.puts "Transfer B finished"
end
IO.puts "Waiting for transfers to be done"
receive do
{:done, ^transfer1} -> nil
end
receive do
{:done, ^transfer2} -> nil
end
send accounts, {:amounts, [:bob, :joe], self}
receive do
{:amounts, amounts} ->
IO.puts "Bob has in account: #{amounts[:bob]}"
IO.puts "Joe has in account: #{amounts[:joe]}"
IO.puts "Inconsistencies while transfer: #{Agent.get(inconsistencies, fn x -> x end)}"
end
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Waiting for transfers to be done
Transfer A started
Transfer B started
Transfer B finished
Transfer A finished
Bob has in account: 100000
Joe has in account: 400000
Inconsistencies while transfer: 0
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We can see that money transfer works without inconsistencies, because we have chosen the message transfer to transfer money and message amounts to get the value of accounts which gives us predictable behavior of the program. Whenever we do a transfer of money, the total amount of money at any time should be the same.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Actor model can cause lock and thus deadlock, so use caution when designing the program. The following script shows how you can simulate the lock and deadlock scenario.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-elixir" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">#
# Deadlock.exs
#
defmodule Lock do
def loop(state) do
receive do
{:lock, sender} ->
case state do
[] ->
send sender, :locked
loop([sender])
_ ->
loop(state ++ [sender])
end
{:unlock, sender} ->
case state do
[] ->
loop(state)
[^sender | []] ->
loop([])
[^sender | [next | tail]] ->
send next, :locked
loop([next | tail])
_ ->
loop(state)
end
end
end
def lock(pid) do
send pid, {:lock, self}
receive do
:locked -> nil # This will block until we receive message
end
end
def unlock(pid) do
send pid, {:unlock, self}
end
def locking(first, second, times) do
if times > 0 do
lock(first)
lock(second)
unlock(second)
unlock(first)
locking(first, second, times - 1)
end
end
end
a_lock = spawn fn -> Lock.loop([]) end
b_lock = spawn fn -> Lock.loop([]) end
this = self
IO.puts "Locking A, B started"
spawn fn ->
Lock.locking(a_lock, b_lock, 1_000)
IO.puts "Locking A, B finished"
send this, :done
end
IO.puts "Locking B, A started"
spawn fn ->
Lock.locking(b_lock, a_lock, 1_000)
IO.puts "Locking B, A finished"
send this, :done
end
IO.puts "Waiting for locking to be done"
receive do
:done -> nil
end
receive do
:done -> nil
End
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script on my laptop I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Locking A, B started
Locking B, A started
Waiting for locking to be done
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
From the output we can see that the processes that lock A and B are stuck. This happens because the first process waits for the second process to release B while second process waiting first process to release A. They are waiting for each other and are stuck forever. To avoid this locking, order should always be the same, or design a program so that it doesn’t use lock (meaning that it doesn’t wait for a specific message). The following listing always locks first A then B.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">#
# Deadlock fixed
#
defmodule Lock do
def loop(state) do
receive do
{:lock, sender} ->
case state do
[] ->
send sender, :locked
loop([sender])
_ ->
loop(state ++ [sender])
end
{:unlock, sender} ->
case state do
[] ->
loop(state)
[^sender | []] ->
loop([])
[^sender | [next | tail]] ->
send next, :locked
loop([next | tail])
_ ->
loop(state)
end
end
end
def lock(pid) do
send pid, {:lock, self}
receive do
:locked -> nil # This will block until we receive message
end
end
def unlock(pid) do
send pid, {:unlock, self}
end
def locking(first, second, times) do
if times > 0 do
lock(first)
lock(second)
unlock(second)
unlock(first)
locking(first, second, times - 1)
end
end
end
a_lock = spawn fn -> Lock.loop([]) end
b_lock = spawn fn -> Lock.loop([]) end
this = self
IO.puts "Locking A, B started"
spawn fn ->
Lock.locking(a_lock, b_lock, 1_000)
IO.puts "Locking A, B finished"
send this, :done
end
IO.puts "Locking A, B started"
spawn fn ->
Lock.locking(a_lock, b_lock, 1_000)
IO.puts "Locking A, B finished"
send this, :done
end
IO.puts "Waiting for locking to be done"
receive do
:done -> nil
end
receive do
:done -> nil
End
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I run this script on my laptop I get:</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 828px;">Locking A, B started
Locking A, B started
Waiting for locking to be done
Locking A, B finished
Locking A, B finished
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And now, there is no longer a deadlock.</div>
<h2 id="wrap-up" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Wrap up</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As an introduction to concurrent programming, we have covered a few concurrency models. We haven’t covered all models, as this article would be too big. Just to name a few, channels and reactive streams are some of the other popularly used concurrency models. Channels and reactive streams have many similarities with the actor model. All of them transmit messages, but many threads can receive messages from one channel, and reactive streams transmit messages in one direction to form directed graph that receive messages from one end and send messages from the other end as a result of the processing.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Shared mutable state models can easily go wrong if we don’t think ahead. It has problems of race condition and deadlock. If we have a choice between different concurrent programming models, it would be easier to implement and maintain but otherwise we have to be very careful what we do.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The functional way is a lot easier to reason about and implement. It cannot have deadlock. This model may have worse performance than shared mutable state model, but a program that works is always faster than one that does not work.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Actor model is a good choice for concurrent programming. Although there are problems of race condition and deadlock, they can happen less than in shared mutable state model since the only way for processes to communicate is via messages. With good message design between processes, that can be avoided. If a problem occurs it is then in the order or meaning of messages in communication between the processes and you know where to look.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I hope this article has given you some insight to what concurrent programming is and how it gives structure to the programs you write.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This article was written by <a class="link is-blue" href="https://www.toptal.com/resume/marko-dvecko" style="border: 0px; box-sizing: border-box; color: #103d77; display: inline; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">MARKO DVEČKO</a>, a <a href="https://www.toptal.com/software/introduction-to-concurrent-programming" target="_blank">Toptal</a> <a href="https://www.toptal.com/developers/all" target="_blank">developer</a>.</div>
Anonymousnoreply@blogger.com5tag:blogger.com,1999:blog-8706416286757464743.post-30931720832116894912016-05-03T14:13:00.000+07:002016-05-03T15:42:42.599+07:00Writing Testable Code in JavaScript: A Brief Overview<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Whether we’re using Node paired with a test framework like Mocha or Jasmine, or spinning up DOM-dependent tests in a headless browser like PhantomJS, our options for unit testing JavaScript are better now than ever.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, this doesn’t mean the code we’re testing is as easy on us as our tools are! Organizing and writing code that is easily testable takes some effort and planning, but there are a few patterns, inspired by functional programming concepts, that we can use to avoid getting into a tough spot when it comes time to test our code. In this article, we will go through some useful tips and patterns for writing testable code in JavaScript.</div>
<h2 id="keep-business-logic-and-display-logic-separate" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
KEEP BUSINESS LOGIC AND DISPLAY LOGIC SEPARATE</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One of the primary jobs of a JavaScript-based browser application is listening to <a href="https://developer.mozilla.org/en-US/docs/Web/Events" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">DOM events</a> triggered by the end user, and then responding to them by running some business logic and displaying the results on the page. It’s tempting to write an anonymous function that does the bulk of the work right where you’re setting up your DOM event listeners. The problem this creates is that you now have to simulate DOM events to test your anonymous function. This can create overhead both in lines of code and the time it takes for tests to run.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Writing Testable Code in JavaScript: A Brief Overview" src="https://assets.toptal.io/uploads/blog/image/92245/toptal-blog-image-1458308379097-cd812b62a07c18b71d42fb0a1956ea48.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Instead, write a named function and pass it to the event handler. That way you can write tests for named functions directly and without jumping through hoops to trigger a fake DOM event.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This applies to more than the DOM though. Many APIs, both in the browser and in Node, are designed around firing and listening to events or waiting for other types of asynchronous work to complete. A rule of thumb is that if you are writing a lot of anonymous callback functions, your code may not be easy to test.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// hard to test</span>
$(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'button'</span>).on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'click'</span>, () => {
$.getJSON(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/path/to/data'</span>)
.then(data => {
$(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'#my-list'</span>).html(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'results: '</span> + data.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">', '</span>));
});
});
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// testable; we can directly run fetchThings to see if it</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// makes an AJAX request without having to trigger DOM</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// events, and we can run showThings directly to see that it</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// displays data in the DOM without doing an AJAX request</span>
$(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'button'</span>).on(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'click'</span>, () => fetchThings(showThings));
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fetchThings</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(callback)</span> {</span>
$.getJSON(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/path/to/data'</span>).then(callback);
}
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">showThings</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(data)</span> {</span>
$(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'#my-list'</span>).html(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'results: '</span> + data.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">', '</span>));
}
</code></pre>
<h2 id="use-callbacks-or-promises-with-asynchronous-code" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
USE CALLBACKS OR PROMISES WITH ASYNCHRONOUS CODE</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the code example above, our refactored fetchThings function runs an <a href="https://en.wikipedia.org/wiki/Ajax_(programming)" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">AJAX</a> request, which does most of its work asynchronously. This means we can’t run the function and test that it did everything we expected, because we won’t know when it’s finished running.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The most common way to solve this problem is to pass a callback function as a parameter to the function that runs asynchronously. In your unit tests you can run your assertions in the callback you pass.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92246/toptal-blog-image-1458308406365-b9e938eb8956e01c8e01ea9b90e6591f.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another common and increasingly popular way to organize asynchronous code is with the Promise API. Fortunately, $.ajax and most other of jQuery’s asynchronous functions return a Promise object already, so a lot of common use cases are already covered.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// hard to test; we don't know how long the AJAX request will run</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fetchData</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
$.ajax({ url: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/path/to/data'</span> });
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// testable; we can pass a callback and run assertions inside it</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fetchDataWithCallback</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(callback)</span> {</span>
$.ajax({
url: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/path/to/data'</span>,
success: callback,
});
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// also testable; we can run assertions when the returned Promise resolves</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">fetchDataWithPromise</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> $.ajax({ url: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'/path/to/data'</span> });
}
</code></pre>
<h2 id="avoid-side-effects" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
AVOID SIDE EFFECTS</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Write functions that take arguments and return a value based solely on those arguments, just like punching numbers into a math equation to get a result. If your function depends on some external state (the properties of a class instance or the contents of a file, for example), and you have to set up that state before testing your function, you have to do more setup in your tests. You’ll have to trust that any other code being run isn’t altering that same state.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92247/toptal-blog-image-1458308451358-80b2558aae716f2965cd6923e2c8dd86.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In the same vein, avoid writing functions that alter external state (like writing to a file or saving values to a database) while it runs. This prevents side effects that could affect your ability to test other code with confidence. In general, it’s best to keep side effects as close to the edges of your code as possible, with as little “surface area” as possible. In case of classes and object instances, a class method’s side effects should be limited to the state of the class instance being tested.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// hard to test; we have to set up a globalListOfCars object and set up a</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// DOM with a #list-of-models node to test this code</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">processCarData</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">()</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> models = globalListOfCars.map(car => car.model);
$(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'#list-of-models'</span>).html(models.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">', '</span>));
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// easy to test; we can pass an argument and test its return value, without</span>
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// setting any global values on the window or checking the DOM the result</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">buildModelsString</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(cars)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">const</span> models = cars.map(car => car.model);
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> models.join(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">','</span>);
}
</code></pre>
<h2 id="use-dependency-injection" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
USE DEPENDENCY INJECTION</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One common pattern for reducing a function’s use of external state is dependency injection - passing all of a function’s external needs as function parameters.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// depends on an external state database connector instance; hard to test</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">updateRow</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(rowId, data)</span> {</span>
myGlobalDatabaseConnector.update(rowId, data);
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// takes a database connector instance in as an argument; easy to test!</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">updateRow</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(rowId, data, databaseConnector)</span> {</span>
databaseConnector.update(rowId, data);
}
</code></pre>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One of the main benefits of using dependency injection is that you can pass in mock objects from your unit tests that don’t cause real side effects (in this case, updating database rows) and you can just assert that your mock object was acted on in the expected way.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #3863a0; line-height: 1.3em;"><b><span style="font-size: medium;">Give Each Function a Single Purpose</span></b></span></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Break long functions that do several things into a collection of short, <a href="https://www.toptal.com/software/single-responsibility-principle" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">single-purpose</a> functions. This makes it far easier to test that each function does its part correctly, rather than hoping that a large one is doing everything correctly before returning a value.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In functional programming, the act of stringing several single-purpose functions together is called composition. Underscore.js even has a function <code style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">_.compose</code>, that takes a list of functions and chains them together, taking the return value of each step and passing it to the next function in line.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// hard to test</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">createGreeting</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(name, location, age)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">let</span> greeting;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (location === <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Mexico'</span>) {
greeting = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'!Hola'</span>;
} <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">else</span> {
greeting = <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Hello'</span>;
}
greeting += <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">' '</span> + name.toUpperCase() + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'! '</span>;
greeting += <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'You are '</span> + age + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">' years old.'</span>;
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> greeting;
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// easy to test</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getBeginning</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(location)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">if</span> (location === <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Mexico'</span>) {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'¡Hola'</span>;
} <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">else</span> {
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'Hello'</span>;
}
}
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getMiddle</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(name)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">' '</span> + name.toUpperCase() + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'! '</span>;
}
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">getEnd</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(age)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'You are '</span> + age + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">' years old.'</span>;
}
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">createGreeting</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(name, location, age)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> getBeginning(location) + getMiddle(name) + getEnd(age);
}
</code></pre>
<h2 id="dont-mutate-parameters" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
DON’T MUTATE PARAMETERS</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In <a href="https://www.toptal.com/javascript" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">JavaScript</a>, arrays and objects are passed by reference rather than value, and they are mutable. This means that when you pass an object or an array as a parameter into a function, both your code and the function you passed the object or array to have the ability to alter the same instance of that array or object in memory. This means that if you’re testing your own code, you have to trust that none of the functions your code calls are altering your objects. Every time you add a new place in your code that alters the same object, it gets increasingly hard to keep track of what that object should look like, making it harder to test.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92248/toptal-blog-image-1458308495515-6a9fce889f1fe99ab8de7fa93c0aebd5.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Instead, if you have a function that takes an object or array have it act on that object or array as though it were read-only. Create a new object or array in code and add values to it based on your needs. Or, use <a href="http://underscorejs.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">Underscore</a>or Lodash to clone the passed object or array before operating on it. Even better, use a tool like Immutable.js that creates read-only data structures.</div>
<pre style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; line-height: 20px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><code class="language-js hljs" style="background: rgb(255, 255, 252); border-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// alters objects passed to it</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">upperCaseLocation</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(customerInfo)</span> {</span>
customerInfo.location = customerInfo.location.toUpperCase();
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> customerInfo;
}
<span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">// sends a new object back instead</span>
<span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">function</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">upperCaseLocation</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(customerInfo)</span> {</span>
<span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> {
name: customerInfo.name,
location: customerInfo.location.toUpperCase(),
age: customerInfo.age
};
}
</code></pre>
<h2 id="write-your-tests-before-your-code" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
WRITE YOUR TESTS BEFORE YOUR CODE</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The process of writing unit tests before the code they’re testing is called <a href="https://en.wikipedia.org/wiki/Test-driven_development" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">test driven development</a> (TDD). A lot of developers find TDD to be very helpful.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
By writing your tests first, you are forced to think about the API you are exposing from the perspective of a developer consuming it. It also helps to ensure you’re only writing enough code to meet the contract being enforced by your tests, rather than over-engineering a solution that’s unnecessarily complex.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In practice, TDD is a discipline that can be difficult to commit to for all your code changes. But when it seems worth trying, it’s a great way to guarantee you are keeping all code testable.</div>
<h2 id="wrap-up" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 14.85px; font-stretch: normal; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
WRAP UP</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We all know there are a few pitfalls that are very easy to fall for when writing and testing complex JavaScript apps. But hopefully with these tips, and remembering to always keep our code as simple and functional as possible, we can keep our test coverage high and overall code complexity low!</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; font-family: 'proxima nova', arial, sans-serif; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<div style="color: #303030; font-size: 1.2em;">
This article was written by <span style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/resume/joshua-mock" style="border: 0px; box-sizing: border-box; color: #103d77; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">JOSHUA MOCK</a></span><span style="color: #929292; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; text-transform: uppercase;"> - FREELANCE SOFTWARE ENGINEER @<a href="https://www.toptal.com/designers/web/writing-testable-code-in-javascript" style="color: blue; text-decoration: none;" target="_blank"> </a></span><span style="color: #3863a0; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif;"><span style="border-color: initial; border-style: initial; box-sizing: border-box; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform;"><a href="https://www.toptal.com/designers/web/writing-testable-code-in-javascript" style="color: blue; text-decoration: none;" target="_blank">TOPTAL</a></span></span></div>
<span style="color: #3863a0; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif;"><br /></span>
<span style="color: #3863a0; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif;">Hire the top 3% of JavaScript developers <a href="https://www.toptal.com/javascript" target="_blank">here</a>.</span></div>
Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-8706416286757464743.post-90878324426311247132016-04-27T00:19:00.001+07:002016-04-27T00:19:41.282+07:00The 5 Most Common UI Design Mistakes<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Although the title <a href="https://www.toptal.com/designers/ui" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;">UI Designer</a> suggests a sort of departure from the traditional <a href="https://www.toptal.com/designers/visual" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;">graphic designer</a>, UI design is still a part of the historical trajectory of the <a href="https://www.toptal.com/designers" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;">visual design</a> discipline.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
With each movement or medium, the discipline has introduced new graphic languages, layouts, and design processes. Between generations, the designer has straddled the transition from press to xerox, or paper to pixel. Across these generations, graphic design has carried out the responsibility of representing the visual language of each era respectively.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Therefore, as UI Design makes the transition out of its infancy, what sort of graphic world can we expect to develop? Unfortunately, based on the current trajectory, the future may look bleak. Much of UI Design today has become standardized and repeatable. Design discussions online involve learning the rules to get designs to safely work, rather than push the envelope, or imagine new things. The tendency for UI Designers to resort to patterns and trends has not only created a bland visual environment, but also diminished the value of the designer as processes become more and more formulaic. The issue is precisely not one of technicalities, but of impending visual boredom.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Thus, the Top Five Common UI Design mistakes are:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; transition: all 0.3s ease-out; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Following Design Rules</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Abusing the Grid</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Misunderstanding Typefaces</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Patterns and the Standardization of UI Design</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Finding Safety in Contrast</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="UI Design Rule Book" src="https://assets.toptal.io/uploads/blog/image/92290/toptal-blog-image-1459323158687-b4ccb3eb448174a9399e58010ae5655a.jpg" style="background: rgb(249, 249, 249); border: 0px; box-shadow: rgba(0, 0, 0, 0.027451) 0px 0px 0px 1px; box-sizing: border-box; display: block; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: all 0.6s ease; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Understand principles and be creative within their properties. Following the rules will only take your where others have been.</div>
<h2 id="common-mistake-1-ui-designers-follow-the-rules" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 25.2px; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMMON MISTAKE #1: UI DESIGNERS FOLLOW THE RULES</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The world of graphic design has always followed sets of rules and standards. Quite often in any design discipline, the common mistakes that are made can closely coincide with a standard rule that has been broken. Thus, from this perspective the design rules seem to be pretty trustworthy to follow.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, in just about any design discipline, new design movements and creative innovation has generally resulted from consciously breaking said rule book. This is possible because design is really conditional, and requires the discretion of the designer, rather than a process with any sort of finite answers. Therefore, the design rules should likely be considered as guidelines more so rather than hard and fast rules. The experienced designer knows and respects the rule book just enough to be able break the box.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Unfortunately, the way that design is often discussed online is within sets of do’s and don’ts. Top mistakes and practices for design in 10 easy steps! Design isn’t so straightforward, and requires a much more robust understanding of principles and tendencies, rather than checklists to systematically carry out.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The concern is that if designers were to cease ‘breaking the rules’, then nothing new creatively would ever be made. If UI designers only develop their ability to follow guidelines, rather than make their own decisions, then they may quickly become irrelevant. How else will we argue a value greater than off the shelf templates?</div>
<h3 id="be-wary-of-top-ten-design-rules" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Be Wary of Top Ten Design Rules</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The issue with design rules in today’s UI design community is they are so abundant. In the interest of solving any problem, the designer can look to the existing UI community and their set of solutions, rather than solve an issue on their own. However, the abundance of these guides and rules have made themselves less credible.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A google search for “Top UI Design Mistakes” yields a half million search results. So, what are the chances that most, if any of these authors of various articles agree with one another? Or, will each design tip that is discussed coincide accurately with the design problems of a reader?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Often the educational articles online discuss acute problems, rather than the guiding design principles behind the issue. The result is that new designers will never learn why design works the way that it does. Instead, they only become able to copy what has come before. Isn’t it concerning that in none of these sorts of articles is something like <a href="http://lust.nl/mobile.php#projects-3465" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">play</a> encouraged?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The designer should have a tool kit of principles to guide them, rather than a book of rules to follow predetermined designs. Press x for parallax scrolling and y for carousels. Before choosing, refer to most recent blog post on which navigational tool is trending. Boring!</div>
<h3 id="tips-and-top-tens-follow-trends" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Tips and Top Tens Follow Trends</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Trends are like junk food for designers. Following trends produces cheap designs that may offer some initial pay back, but little worth in the long run. This means that not only may trendy designers become dated, or ineffective quickly. But, for you the designer, don’t expect to experience any sense of reward when designing in this way. Although working to invent your own styles and systems is a lot of work, it’s so worth it day in and day out. There’s just something about copying that never seems to feed the soul.</div>
<h2 id="common-mistake-2-allowing-the-grid-to-restrict-ui-design" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 25.2px; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMMON MISTAKE #2: ALLOWING THE GRID TO RESTRICT UI DESIGN</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Despite my treatise against rules - here’s a rule: there is no way for a UI Designer to design without a grid. The web or mobile interface is fundamentally based on a pixel by pixel organization - there’s no way around it. However, this does not necessarily mean that the interface has to restrict designers to gridded appearances, or even gridded processes.</div>
<h3 id="using-the-grid-as-a-trendy-tool" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Using the Grid as a Trendy Tool</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Generally, making any design moves as a response to trends can easily lead to poor design. Perhaps what results is a satisfactory, mostly functional product. But it will almost certainly be boring or uninteresting. To be trendy is to be commonplace. Therefore, when employing the grid in a design, understand what the grid has to offer as a tool, and what it might convey. Grids generally represent neutrality, as everything within the restraints of a grid appear equal. Grids also allow for a neutral navigational experience. Users can jump from item to item without any interference from the designer’s curatorial hand. Whereas, with other navigational structures, the designer may be able to group content, or establish desired sequences.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="UI Design Rule Book" src="https://assets.toptal.io/uploads/blog/image/92291/toptal-blog-image-1459323274112-249e7c8e02dfe49d991dd95dcf8f9906.jpg" style="background: rgb(249, 249, 249); border: 0px; box-shadow: rgba(0, 0, 0, 0.027451) 0px 0px 0px 1px; box-sizing: border-box; display: block; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: all 0.6s ease; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Although a useful tool, the grid can be very limiting to designers.</div>
<h3 id="defaulting-to-the-grid-as-a-work-flow" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Defaulting to the Grid as a Work Flow</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://htmloutput.risd.gd/interview-mevis/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">Dylan Fracareta</a>, faculty of RISD and director of PIN-UP Magazine, points out that “most people start off with a 12 - column grid…because you can get 3 and 4 off of that”. The danger here is that immediately the designer predetermines anything that they might come up with. Alternatively, Fracareta resides to only using the move tool with set quantities, rather than physically placing things against a grid line. Although this establishes order, it opens up more potential for unexpected outcomes. Although designing for the browser used to mean that we would input some code, wait, and see what happens. Now, web design has returned to a more traditional form of layout designer that’s “more like adjusting two sheets of transparent paper”. How can we as designers benefit from this process? Working Without a Grid Although grids can be restricting, they are one of our most traditional forms of <a href="http://www.graphics.com/article-old/brief-history-grids" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">organization</a>. The grid is intuitive. The grid is neutral and unassuming. Therefore, grids allow content to speak for itself, and for users to navigate at their will and with ease. Despite my warnings towards the restrictiveness of grids, different arrays allow for different levels of guidance or freedom.</div>
<h2 id="common-mistake-3the-standardization-of-ui-design-with-patterns" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 25.2px; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMMON MISTAKE #3:THE STANDARDIZATION OF UI DESIGN WITH PATTERNS</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The concept of standardized design elements predates UI design. Architectural details have been frequently repeated in practice for typical conditions for centuries. Generally this practice makes sense for certain parts of a building that are rarely perceived by a user. However, once architects began to standardize common elements like furniture dimensions, or handrails heights, people eventually expressed disinterest in the boring, beige physical environment that resulted. Not only this, but standardized dimensions were proven to be ineffective, as although generated as an average, they didn’t really apply to the majority of the population. Thus, although repeatable detail have their place, they should be used critically.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="If we as designers choose to automate, what value are we providing?" src="https://assets.toptal.io/uploads/blog/image/92292/toptal-blog-image-1459342543880-93071dd9ca327a5e6102c7e2d9120cad.jpg" style="background: rgb(249, 249, 249); border: 0px; box-shadow: rgba(0, 0, 0, 0.027451) 0px 0px 0px 1px; box-sizing: border-box; display: block; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: all 0.6s ease; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
If we as designers choose to automate, what value are we providing?</div>
<h3 id="designers-using-the-pattern-as-product" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Designers Using the Pattern as Product</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Many UI designers don’t view the pattern as a time saving tool, but rather an off the shelf solution to design problems. Patterns are intended to take recurring tasks or artefacts and standardize them in order to make the designer’s job easier. Instead, certain patterns like F Pattern Layouts, Carousels or Pagination have become the entire structure of many of our interfaces.</div>
<h3 id="justification-for-the-pattern-is-skewed" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Justification for the Pattern is Skewed</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Designers tell themselves that the F shaped pattern exists as a result of the way that people read on the web. <a href="https://www.smashingmagazine.com/2016/01/is-the-internet-killing-creativity/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">Espen Brunborg</a> points out that perhaps people read this way as a result of us designing for that pattern. “What’s the point of having web designers if all they do is follow the recipe,” Brunborg asks.</div>
<h2 id="common-mistake-4-misunderstanding-typefaces" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 25.2px; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMMON MISTAKE #4: MISUNDERSTANDING TYPEFACES</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Many designer’s quick tips suggest hard and fast rules about fonts as well. Each rule is shouted religiously, “One font family only! <a href="http://practicaltypography.com/monospaced-fonts.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">Monospaced</a> fonts are dead! Avoid thin fonts at all costs!”. But really, the only legitimate rules on type, text and fonts should be to enforce legibility, and convey meaning. As long as type is legible, there may very well be an appropriate opportunity for all sorts of typefaces. The UI Designer must take on the responsibility of knowing the history, uses, and designed intentions for each font that they implement in a UI.</div>
<h3 id="consider-a-typeface-only-for-legibility" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Consider a Typeface Only for Legibility</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Typefaces convey meaning as well as affect legibility. With all of the discussion surrounding rules for proper legibility on devices etc, designers are forgetting that type is designed to augment a body of text with a sensibility, as much as it is meant to be legible. Legibility is critical, I do not dispute this - but my point is that legibility really should be an obvious goal. Otherwise, why wouldn’t we have just stopped at Helvetica, or maybe <a href="http://www.citylab.com/commute/2016/01/official-united-states-highway-sign-font-clearview/427068/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">Highway Gothic</a>. However, the important thing to remember is that fonts are not just designed for different contexts of legibility. Typefaces are also essential for conveying meaning or giving a body of text a mood.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92293/toptal-blog-image-1459342565358-fc74a7cf1c7decc2af12b6266a9ebf73.jpg" style="background: rgb(249, 249, 249); border: 0px; box-shadow: rgba(0, 0, 0, 0.027451) 0px 0px 0px 1px; box-sizing: border-box; display: block; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; opacity: 1; padding: 0px; transition: all 0.6s ease; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Typefaces are each designed for their own uses. Don't allow narrow minded rules to restrict an exploration of the world of type.</div>
<h3 id="avoiding-thin-fonts-at-all-costs" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Avoiding Thin Fonts At All Costs</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now that the trend has come (and almost gone?), a common design criticism is to avoid thin fonts entirely. In the same way thin fonts came as a trend, they may leave as one also. However, the hope should be to understand the principles of the typefaces rather than follow trends at all.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Some say that they’re impossible to read or untrustworthy between devices. All legitimate points. Yet, this represents a condition in the current discussion of UI design. The font choice is only understood by designers as technical choice in regards to legibility, rather than also understanding the meaning and value of typefaces. The concern is that if legibility is the only concern that a designer carried, would thin fonts be done away with entirely?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Understand why you are using a thin font, and within what contexts. Bold, thick text is actually much more difficult to read at length than thinner fonts. Yet, as bold fonts carry more visual weight they’re more appropriate for headings, or content with little text. As thin fonts are often serifs, its suitability for body text is entirely objective. As serif characters flow together when read in rapid succession, they make for much more comfortable long reading.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As well, thin fonts are often chosen because they convey elegance. So, if a designer was working on an interface for a client whose mandate was to convey elegance, they might find themselves hard pressed to find a heavy typeface to do the job.</div>
<h3 id="not-enough-variation" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Not Enough Variation</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A common mistake is to not provide enough variation between fonts in an interface. Changing fonts is a good navigational tool to establish visual hierarchy, or potentially different functions within an interface. A crash course on hierarchy will teach you that generally the largest items, or boldest fonts, should be the most important, and carry the most visual weight. Visual importance can convey content headings, or perhaps frequently used functions.</div>
<h3 id="too-much-variation" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Too Much Variation</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A common UI Design mistake is to load in several different typefaces from different families that each denote a unique function. The issue with making every font choice special, when there is many fonts, is that no font stands out. Changing fonts is a good navigational tool to establish visual hierarchy, or potentially different functions within an interface. Therefore, if every font is different, there is too much confusion for a user to recognize any order.</div>
<h2 id="common-mistake-5-underover-estimating-the-potential-of-contrast" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 25.2px; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; text-transform: uppercase; vertical-align: baseline;">
COMMON MISTAKE #5: UNDER/OVER ESTIMATING THE POTENTIAL OF CONTRAST</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A common mistake that appears on many Top UI Design Mistake lists is that designers should avoid low contrast interfaces. There are many instances in which low contrast designs are illegible and ineffective - true. However, as with the previous points, my worry is that this use of language alternatively produces a high contrast design culture in response.</div>
<h3 id="defaulting-to-high-contrast" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Defaulting to High Contrast</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The issue is that high contrast is aesthetically easy to achieve. High contrast visuals are undeniably stimulating or exciting. However, there are many more moods in the human imagination to convey or communicate with, other than high stimulation. To be visually stimulating may also be visually safe.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The same issue is actually occurring in <a href="http://screenrobot.com/samurai-jack-movie-modern-sci-fi-needs/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;" target="_blank">sci-fi</a> film. The entire industry has resorted to black and neon blue visuals as a way to trick viewers into accepting ‘exciting’ visuals, instead of new, creative, or beautiful visuals. This article points out what the sci-fi industry is missing out on by producing safe visuals.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Functionally, if every element in an interface is in high contrast to another, then nothing stands out. This defeats the potential value of contrast as a hierarchical tool. Considering different design moves as tools, rather than rules to follow is essential in avoiding stagnant, trendy design.</div>
<h3 id="illegibly-low-contrast" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Illegibly Low Contrast</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The use of low contrast fonts and backgrounds is a commonly made mistake. However, rather than being a design issue. This could potentially be discussed as a beta testing mistake, rather than a design mistake.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How the design element relates as a low contrast piece to the rest of the interface is a design concern. The issue could be that the most significant item hierarchically is low in contrast to the rest of the interface. For the interface to communicate its organizational structure, the elements should contrast one another in a certain way. This is a design discussion. Whether or not it is legible is arguably a testing mistake.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The point is that in only discussing contrast as a technical issue resolvable by adjusting a value, designers miss out on the critical understanding of what contrast is principally used for.</div>
<h3 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; font-stretch: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; position: relative; vertical-align: baseline;">
Conclusion</h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As with the previous 4 mistakes, the abuse of patterns will rarely result in a dysfunctional website, but rather just a boring one. The mistake is in being safe. This overly cautious method of design may not cause the individual project to fail. However, this series of safe mistakes performed by the greater web community can mean greater failures beyond the individual UI design project. The role of the designer should be to imagine, thoughtfully experiment and create - not to responsibly follow rules and guidelines.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; font-stretch: inherit; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The original article is found on the <a href="https://www.toptal.com/designers/ui/most-common-ui-design-mistakes" style="border: 0px; box-sizing: border-box; color: #637182; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: none; padding: 0px; text-decoration: none; transition: all 0.25s; vertical-align: baseline;">Toptal Design Blog</a>.</div>
Anonymousnoreply@blogger.com3tag:blogger.com,1999:blog-8706416286757464743.post-10285863345294115442016-04-18T22:10:00.001+07:002016-04-18T22:10:16.722+07:00Top Ten Front-End Design Rules For Developers<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
As front-end developers, our job is, essentially, to turn designs into reality via code. Understanding, and being competent in, design is an important component of that. Unfortunately, truly understanding front-end design is easier said than done. Coding and aesthetic design require some pretty different skill sets. Because of that, some front-end devs aren’t as proficient in the design aspect as they should be, and as a result, their work suffers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
My goal is to give you some easy-to-follow rules and concepts, from one <a href="https://www.toptal.com/front-end" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;">front-end dev</a> to another, that will help you go from start to finish of a project without messing up what your designers worked so hard on (or possibly even allowing you to design your own projects with decent results).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Of course, these rules won’t take you from bad to magnificent in the time it takes to read one article, but if you apply them to your work, they should make a big difference.</div>
<h2 id="do-stuff-in-a-graphics-program" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Do Stuff In A Graphics Program</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
It’s truly rare that you complete a project, and go from start to finish while maintaining every single aesthetic mutation in the design files. And, unfortunately, designers aren’t always around to run to for a quick fix.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Therefore, there always comes a point in any front-end job where you end up having to make some aesthetic-related tweaks. Whether it’s making the checkmark that shows when you check the checkbox, or making a page layout that the PSD missed, front-enders often end up handling these seemingly minor tasks. Naturally, in a perfect world this wouldn’t be the case, but I have yet to find a perfect world, hence we need to be flexible.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="A good front-end developer has to use professional graphics tools. Accept no substitute." src="https://assets.toptal.io/uploads/blog/image/92264/toptal-blog-image-1458716073387-395562414254485592387fc148bdf770.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
A good front-end developer has to use professional graphics tools. Accept no substitute.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
For these situations, you should always use a graphics program for mockups. I don’t care which tool you choose: Photoshop, Illustrator, Fireworks, <a href="https://www.gimp.org/tutorials/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;" target="_blank">GIMP</a>, whatever. Just don’t just attempt to design from your code. Spend a minute launching a real graphics program and figuring out how it should look, then go to the code and make it happen. You may not be an expert designer, but you’ll still end up with better results.</div>
<h2 id="match-the-design-dont-try-to-beat-it" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Match the Design, Don’t Try To Beat It</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Your job is not to impress with how unique your checkmark is; your job is to match it to the rest of the design.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Those without a lot of design experience can easily be tempted to leave their mark on the project with seemingly minor details. Please leave that to the designers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Developers have to match the original front-end design as closely as possible." src="https://assets.toptal.io/uploads/blog/image/92265/toptal-blog-image-1458716095716-49feb67ac92ee285b2947b9faefd8134.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Developers have to match the original front-end design as closely as possible.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Instead of asking “Does my checkmark look amazing?” you should be asking, “How well does my checkmark match the design?”</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Your focus should always be on working with the design, not on trying to outdo it.</div>
<h2 id="typography-makes-all-the-difference" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Typography Makes All the Difference</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
You’d be surprised to know how much of the end look of a design is influenced by typography. You’d be just as surprised to learn how much time designers spend on it. This is not a “pick-it-and-go” endeavor, some serious time and effort goes into it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
If you end up in a situation where you actually have to choose typography, you should spend a decent amount of time doing so. Go online and research good <a href="https://femmebot.github.io/google-type/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;" target="_blank">font pairings</a>. Spend a few hours trying those pairings and making sure you end up with the best typography for the project.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Is this font right for your project? When in doubt, consult a designer." src="https://assets.toptal.io/uploads/blog/image/92266/toptal-blog-image-1458716109882-5d65c1fd00a173216174c77f96d13b79.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Is this font right for your project? When in doubt, consult a designer.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
If you’re working with a design, then make sure you follow the designer’s typography choices. This doesn’t just mean choosing the font, either. Pay attention to the line spacing, letter spacing, and so on. Don’t overlook how important it is to match the typography of the design.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Also, make sure you use the right fonts in the correct spot. If the designer uses Georgia for headers only and Open Sans for body, then you shouldn’t be using Georgia for body and Open Sans for headers. Typography can make or break aesthetics easily. Spend enough time making sure you are matching your designer’s typography. It will be time well spent.</div>
<h2 id="front-end-design-doesnt-tolerate-tunnel-vision" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Front-end Design Doesn’t Tolerate Tunnel Vision</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
You’ll probably be making small parts of the overall design.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Tunnel vision is a common pitfall for front-end developers. Don’t focus on a single detail, always look at the big picture." src="https://assets.toptal.io/uploads/blog/image/92267/toptal-blog-image-1458716122135-f87f3809e75ccdfbd62b665248d99279.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Tunnel vision is a common pitfall for front-end developers. Don’t focus on a single detail, always look at the big picture.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
An example I’ve been going with is making the checkmark for a design that includes custom checkboxes, without showing them checked. It’s important to remember that the parts you are making are small parts of an overall design. Make your checks as important as a checkmark on a page should look, no more, no less. Don’t get tunnel vision about your one little part and make it something it shouldn’t be.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
In fact, a good technique for doing this is to take a screenshot of the program so far, or of the design files, and design within it, in the context in which it will be used. That way, you really see how it affects other design elements on the page, and whether it fits its role properly.</div>
<h2 id="relationships-and-hierarchy" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Relationships And Hierarchy</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Pay special attention to how the design works with <a href="https://hackdesign.org/lessons/19" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;" target="_blank">hierarchy</a>. How close are the titles to the body of text? How far are they from the text above them? How does the designer seem to be indicating which elements/titles/text bodies are related and which aren’t? They’ll commonly do these things by boxing related content together, using varying white space to indicate relationships, using similar or contrasting colors to indicate related/unrelated content, and so on.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="A good front-end developer will respect design relationships and hierarchy. A great developer will understand them." src="https://assets.toptal.io/uploads/blog/image/92268/toptal-blog-image-1458716143751-c8407153d29163186c28bf7a7e392499.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
A good front-end developer will respect design relationships and hierarchy. A great developer will understand them.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
It’s your job to make sure that you recognize the ways in which the design accomplishes relationships and hierarchy and to make sure those concepts are reflected in the end product (including for content that was not specifically designed, and/or dynamic content). This is another area (like typography) where it pays to take extra time to make sure you’re doing a good job.</div>
<h2 id="be-picky-about-whitespace-and-alignment" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Be Picky About Whitespace And Alignment</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
This is a great tip for improving your designs and/or better implementing the designs of others: If the design seems to be using spacings of 20 units, 40 units, etc., then make sure every spacing is a multiple of 20 units.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
This is a really drop-dead simple way for someone with no eye for aesthetics to make a significant improvement quickly. Make sure your elements are aligned down to the pixel, and that the spacing around every edge of every element is as uniform as possible. Where you can’t do that (such as places where you need extra space to indicate hierarchy), make them exact multiples of the spacing you’re using elsewhere, for example two times your default to create some separation, three times to create more, and so on.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Do your best to understand how the designer used whitespace and follow those concepts in your front-end build." src="https://assets.toptal.io/uploads/blog/image/92269/toptal-blog-image-1458716159330-92776c29f68e372c6ed5e5ebd516747f.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Do your best to understand how the designer used whitespace and follow those concepts in your front-end build.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
A lot of devs achieve this for specific content in the design files, but when it comes to adding/editing content, or implementing dynamic content, the spacing can go all over the place because they didn’t truly understand what they were implementing.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Do your best to understand how the designer used whitespace and follow those concepts in your build. And yes, spend time on this. Once you think your work is done, go back and measure the spacing to ensure you have <em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">aligned and uniformly spaced</em> everything as much as possible, then try out the code with lots of varying content to make sure <em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">it’s flexible</em>.</div>
<h2 id="if-you-dont-know-what-youre-doing-do-less" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
If You Don’t Know What You’re Doing, Do Less</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
I’m not one of those people that thinks every project should use minimalist design, but if you’re not confident in your design chops and you need to add something, then less is more.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Less is more. If your designer did a good job to begin with, you should refrain from injecting your own design ideas." src="https://assets.toptal.io/uploads/blog/image/92270/toptal-blog-image-1458716184767-659686e7ecac998937dc61b89c2cd45c.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Less is more. If your designer did a good job to begin with, you should refrain from injecting your own design ideas.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
The designer took care of the main stuff; you only need to do minor fillers. If you’re not very good at design, then a good bet is to do as minimal amount as you can to make that element work. That way, you’re injecting less of your own design into the designer’s work, and affecting it as little as possible.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Let the designer’s work take center stage and let your work take the back seat.</div>
<h2 id="time-makes-fools-of-us-all" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Time Makes Fools Of Us All</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
I’ll tell you a secret about designers: 90 percent (or more) of what they actually put down on paper, or a Photoshop canvas, isn’t that great.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
They discard far more than you ever see. It often takes many revisions and fiddling with a design to get it to the point where they’d even let the guy in the next cubicle see their work, never mind the actual client. You usually don’t go from a blank canvas to good design in one step; there’s a bunch iterations in between. People rarely make good work until they understand that and allow for it in their process.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="If you think the design can be improved upon, consult your designer. It’s possible they already tried a similar approach and decided against it." src="https://assets.toptal.io/uploads/blog/image/92271/toptal-blog-image-1458716216764-ac92e0ff456ef4d1285200f7ea5f8b89.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
If you think the design can be improved upon, consult your designer. It’s possible they already tried a similar approach and decided against it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
So how do you implement this? One important method is taking time between versions. Work until it looks like something you like then put it away. Give it a few hours (leaving it overnight is even better), then open it up again and take a look. You’ll be amazed at how different it looks with fresh eyes. You’ll quickly pick out areas for improvement. They’ll be so clear you’ll wonder how you possibly missed them in the first place.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
In fact, one of the better designers I’ve known takes this idea a lot further. He would start by making three different designs. Then, he’d wait at least 24 hours, look at them again and throw them all out and start from scratch on a fourth. Next, he’d allow a day between each iteration as it got better and better. Only when he opened it up one morning, and was totally happy, or at least, as close as a designer ever gets to totally happy, would he send it to the client. This was the process he used for every design he made, and it served him very well.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
I don’t expect you to take it <em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">that</em> far, but it does highlight how helpful time without “eyes on the design” can be. It’s an integral part of the design process and can make improvements in leaps and bounds.</div>
<h2 id="pixels-matter" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Pixels Matter</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
You should do everything in your power to match the original design in your finished program, down to the last pixel.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Front-end developers should try to match the original design down to the last pixel." src="https://assets.toptal.io/uploads/blog/image/92272/toptal-blog-image-1458716242193-f494f220b24e29ee6c07289f9d745a2d.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Front-end developers should try to match the original design down to the last pixel.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
In some areas you can’t be perfect. For example, your control over letter-spacing might not be quite as precise as that of the designer’s, and a CSS shadow might not exactly match a Photoshop one, but you should still attempt to get as close as possible. For many aspects of the design, you really can get pixel-perfect precision. Doing so can make a big difference in the end result. A pixel off here and there doesn’t seem like much, but it adds up and affects the overall aesthetic much more than you’d think. So keep an eye on it.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
There are a number of [tools] that help you compare original designs to end results, or you can just take screenshots and paste them into the design file to compare each element as closely as possible. Just lay the screenshot over the design and make it semi-transparent so that you can see the differences. Then you know how much adjustment you have to make to get it spot on.</div>
<h2 id="get-feedback" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Get Feedback</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
It’s hard to gain an “eye for design.” It’s even harder to do it on your own. You should seek the <a href="http://graphicdesign.stackexchange.com/questions/19927/where-can-i-go-to-recieve-feedback-on-web-design-projects" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;" target="_blank">input of others</a>to really see how you can make improvements.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
I am not suggesting you grab your neighbor and ask for advice, I mean you should consult real designers and let them critique your work and offer suggestions.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Let designers critique your work. Put their criticism to good use and don’t antagonize them." src="https://assets.toptal.io/uploads/blog/image/92274/toptal-blog-image-1458716950005-84068405372c40feaeb9e5e268dc8825.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Let designers critique your work. Put their criticism to good use and don’t antagonize them.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
It takes some bravery to do so, but in the end it is one of the most powerful things you can do to improve the project in the short-term, and to improve your skill level in the long run.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Even if all you have to fine tune is a simple checkmark, there are plenty of people willing to help you. Whether it’s a designer friend, or an online forum, seek out qualified people and get their feedback.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Build a long-lasting, productive relationship with your designers. It’s vital for useful feedback, quality, and execution." src="https://assets.toptal.io/uploads/blog/image/92273/toptal-blog-image-1458716933688-80136bb5879f73a3bca8a9ad8b9f275e.jpg" style="border: 0px; box-sizing: border-box; display: block; height: auto; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; outline: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Build a long-lasting, productive relationship with your designers. It’s vital for useful feedback, quality, and execution.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
It may sound time consuming, and may cause friction between you and your designers, but in the big scheme of things, it’s worth it. Good front-end developers rely on valuable input from designers, even when it’s not something they like to hear.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Therefore, it’s vital to build and maintain a constructive relationship with your designers. You’re all in the same boat, so to get the best possible results you have to collaborate and communicate every step of the way. The investment in building bonds with your designers is well worth it, as it will help everyone do a better job and execute everything on time.</div>
<h2 id="conclusion" style="background-color: white; border: 0px; box-sizing: border-box; clear: both; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 32px; font-weight: inherit; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Conclusion</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
To summarize, here is a short list of design tips for front-end developers:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Design in a graphics program. Don’t design from code, not even the small stuff.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Match the design. Be conscious of the original design and don’t try to improve it, just match it.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Typography is huge. The time you spend making sure it’s right should reflect its importance.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Avoid tunnel vision. Make sure your additions stand out only as much as they should. They’re not more important just because you designed them.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Relationships and hierarchy: Understand <em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">how</em> they work in the design so that you can implement them properly.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Whitespace and alignment are important. Make them accurate to the pixel and make them evenly throughout anything you add.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">If you’re not confident in your skills, then make your additions as minimally styled as you can.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Take time between revisions. Come back later to see your design work with fresh eyes.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Pixel-perfect implementation is important wherever possible.</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Be brave. Seek out experienced designers to critique your work.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
Not every front-end developer is going to be a fantastic designer, but every front-end dev should at least be<em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">competent</em> in terms of design.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
You need to understand enough about design concepts to identify what’s going on, and to properly apply the design to your end product. Sometimes, you can get away with blind copying if you’ve got a thorough designer (and if you’re detail oriented enough to truly copy it pixel for pixel).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
However, in order to make large projects shine across many variations of content, you need some understanding of what’s going through the designer’s head. You don’t merely need to see what the design looks like, you need to know <em style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 21.6px; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">why it looks the way it does</em>, and that way you can be mindful of technical and aesthetic limitations that will affect your job.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
So, even as a front-end developer, part of your regular self-improvement should always include learning more about design.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
The original article was written by <span style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-style: inherit; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/resume/bryan-grezeszak" style="border: 0px; box-sizing: border-box; color: #103d77; display: inline; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">BRYAN GREZESZAK</a></span><span style="border: 0px; box-sizing: border-box; color: #929292; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-style: inherit; font-weight: inherit; letter-spacing: 0.13px; line-height: 19px; margin: 0px; outline: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"> - FREELANCE SOFTWARE ENGINEER @ </span><a class="link is-blue" href="https://www.toptal.com/" style="border: 0px; box-sizing: border-box; color: #3863a0; display: inline; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-style: inherit; font-weight: inherit; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; outline: 0px; padding: 0px; text-decoration: none; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">TOPTAL</a> and can be read <a href="https://www.toptal.com/designers/app/front-end-design-principles" style="border: 0px; box-sizing: border-box; color: #f45145; font-family: inherit; font-size: 21.6px; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; transition: all 0.2s ease-in-out; vertical-align: baseline;" target="_blank">here</a>.</div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-19968638914940452032016-04-13T23:28:00.003+07:002016-04-13T23:28:47.643+07:00HSA For Developers: Heterogeneous Computing For The Masses<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What do chipmakers like AMD, ARM, Samsung, MediaTek, Qualcomm, and Texas Instruments have in common? Well, apart from the obvious similarities between these chip-making behemoths, they also happen to be founders of the HSA Foundation. What’s HSA, and why does it need a foundation backed by industry heavyweights?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In this post I will try to explain why HSA could be a big deal in the near future, so I’ll start with the basics: <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">What is HSA and why should you care</em>?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
HSA stands for Heterogeneous System Architecture, which sounds kind of boring, but trust me, it could become very exciting, indeed. HSA is essentially a set of standards and specifications designed to allow further integration of CPUs and GPUs on the same bus. This is not an entirely new concept; desktop CPUs and mobile SoCs have been employing integrated graphics and using a single bus for years, but HSA takes it to the next level.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Same load, different architectures: CPUs and GPUs excel at different tasks. What happens when they start sharing the load, with no developer input?" src="https://assets.toptal.io/uploads/blog/image/92337/toptal-blog-image-1460094829029-96103e317384b9c4694e2b6fe7d69bb9.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Same load, different architectures: CPUs and GPUs excel at different tasks. What happens when they start sharing the load, with no developer input?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Rather than simply using the same bus and shared memory for the CPU and GPU, HSA also allows these two vastly different architectures to work in tandem and <span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">share tasks</span>. It might not sound like a big deal, but if you take a closer look, and examine the potential long-term effects of this approach, it starts to look very “sweet” in a technical sense.</div>
<h2 id="oh-no-heres-another-silly-standard-developers-have-to-implement" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Oh No! Here’s Another Silly Standard Developers Have To Implement</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Yes and no.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The idea of sharing the same bus is not new, and neither is the idea of employing highly parallelised GPUs for certain compute tasks (which don’t involve rendering headshots). It’s been done before, and I guess most of our readers are already familiar with GPGPU standards like <a href="https://www.toptal.com/cuda" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">CUDA</a> and OpenCL.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, unlike the CUDA or OpenCL approach, HSA would effectively take the developer out of the equation, at least when it comes to assigning different loads to different processing cores. The hardware would decide when to offload calculations from the CPU to the GPU and vice versa. HSA is not supposed to replace established GPGPU programming languages like OpenCL, as they can be implemented on HSA hardware as well.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
That’s the whole point of HSA: <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">It’s supposed to make the whole process easy, even seamless.</em> Developers won’t necessarily have to think about offloading calculations to the GPU. The hardware will do it automatically.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="A lot of big names support HSA. However, industry heavyweights Intel and Nvidia are not on the list." src="https://assets.toptal.io/uploads/blog/image/92338/toptal-blog-image-1460094883546-fb182a22671fe9ae128250632107196e.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
A lot of big names support HSA. However, industry heavyweights Intel and Nvidia are not on the list.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To accomplish this, HSA will have to enjoy support from multiple chipmakers and hardware vendors. While the list of HSA supporters is impressive, Intel is conspicuously absent from this veritable who’s who of the chip industry. Given Intel’s market share in both desktop and server processor markets, this is a <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">big</em> deal. Another name you won’t find on the list is Nvidia, which is focused on CUDA, and is currently the GPU compute market leader.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
However, HSA is not designed solely for high performance systems and applications, on hardware that usually sports an <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Intel Inside</em> sticker. HSA can also be used in energy efficient mobile devices, where Intel has a negligible market share.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So, HSA is supposed to make life easier, but is it relevant yet? Will it catch on? This is not technological question, but an economic one. It will depend on the invisible hand of the market. So, before we proceed, let’s start by taking a closer look at where things stand right now, and how we got here.</div>
<h2 id="hsa-development-teething-problems-and-adoption-concerns" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
HSA Development, Teething Problems And Adoption Concerns</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As I said in the introduction, HSA is not exactly a novel concept. It was originally envisioned by Advanced Micro Devices (AMD), which had a vested interest in getting it off the ground. A decade ago, AMD bought graphics specialists ATI, and since then the company has been trying to leverage its access to cutting edge GPU technology to boost overall sales.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On the face of it, the idea was simple enough: AMD would not only continue developing and manufacturing cutting-edge discrete GPUs, it would also integrate ATI’s GPU technology in its processors. AMD’s marketing department called the idea ‘Fusion’, and HSA was referred to as Fusion System Architecture (FSA). Sounds great, right? Getting a decent x86 processor with good integrated graphics sounded like a good idea, and it was.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Unfortunately, AMD ran into a number of issues along the way; I’ll single out a few of them:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Any good idea in tech is bound to be picked up by competitors, in this case – Intel.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">AMD lost the technological edge to Intel and found it increasingly difficult to compete in the CPU market due to Intel’s foundry technology lead.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">AMD’s execution was problematic and many of the new processors were late to market. Others were scrapped entirely.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">The economic meltdown of 2008 and subsequent mobile revolution did not help.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
These, and a number of other factors, conspired to blunt AMD’s edge and prevent market adoption of its products and technologies. AMD started rolling out processors with the new generation of integrated Radeon graphics in mid-2011, and it started calling them Accelerated Processing Units (APUs) instead of CPUs.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Marketing aside, AMD’s first generation of APUs (codenamed Llano), was a flop. The chips were late and could not keep up with Intel’s offerings. Serious HSA features were not included either, but AMD started adding them in its 2012 platform (Trinity, which was essentially Llano done right). The next step came in 2014, with the introduction of Kaveri APUs, which supported heterogeneous memory management (the GPU IOMMU and CPU MMU shared the same address space). Kaveri also brought about more architectural integration, enabling coherent memory between the CPU and GPU (AMD calls it <a href="http://www.amd.com/en-us/innovations/software-technologies/compute-cores" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">hUMA</a>, which stands for Heterogeneous Unified Memory Access) . The subsequent Carizzo refresh added even more HSA features, enabling the processor to context switch compute tasks on the GPU and do a few more tricks.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The upcoming Zen CPU architecture, and the APUs built on top of it, promises to deliver even more, if and when it shows up on the market.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
So what’s the problem?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
AMD was not the only chipmaker to realise the potential of on-die GPUs. Intel started adding them to its Core CPUs as well, as did ARM chipmakers, so integrated GPUs are currently used in virtually every smartphone SoC, plus the vast majority of PCs/Macs. In the meantime, AMD’s position in the CPU market was eroded. The market share slump made AMD’s platforms less appealing to developers, businesses, and even consumers. There simply aren’t that many AMD-based PCs on the market, and Apple does not use AMD processors at all (although it did use AMD graphics, mainly due to OpenCL compatibility).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
AMD no longer competes with Intel in the high-end CPU market, but even if it did, it wouldn’t make much of a difference in this respect. People don’t buy $2,000 workstations or gaming PCs to use integrated graphics. They use pricey, discrete graphics, and don’t care much about energy efficiency.</div>
<h2 id="how-about-some-hsa-for-smartphones-and-tablets" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How About Some HSA For Smartphones And Tablets?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But, wait. What about mobile platforms? Couldn’t AMD just roll out similar solutions for smartphone and tablet chips? Well, no, not really.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You see, a few years after the ATI acquisition, AMD found itself in a tough financial situation, compounded by the economic crisis, so it decided to sell off its Imageon mobile GPU division to Qualcomm. Qualcomm renamed the products Adreno (anagram of Radeon), and went on to become the dominant player in the smartphone processor market, using freshly repainted in-house GPUs.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As some of you may notice, selling a smartphone graphics outfit just as the smartphone revolution was about to kick off, does not look like a brilliant business move, but I guess hindsight is always 20/20.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
HSA used to be associated solely with AMD and its x86 processors, but this is no longer the case. In fact, if all HSA Foundation members started shipping HSA-enabled ARM smartphone processors, they would outsell AMD’s x86 processors several fold, both in terms of revenue and units shipped. So what happens if they do? What would that mean for the industry and developers?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Well, for starters, smartphone processors already rely on heterogeneous computing, sort of. Heterogeneous computing usually refers to the concept of using different architectures in a single chip, and considering all the components found on today’s highly integrated SoCs, this could be a very broad definition. As a result, nearly every SoC may be considered a heterogeneous computing platform, depending on one’s standards. Sometimes, people even refer to different processors based on the same instruction set as a heterogeneous platform (for example, mobile chips with ARM Cortex-A57 and A53 cores, both of which are based on the 64-bit ARMv8 instruction set).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Many observers agree that most ARM-based processors may now be considered heterogeneous platforms, including Apple A-series chips, Samsung Exynos SoCs and similar processors from other vendors, namely big players like Qualcomm and MediaTek.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But why would anyone <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">need</em> HSA on smartphone processors? Isn’t the whole point of using GPUs for general computing to deal with professional workloads, not Angry Birds and Uber?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Yes it is, but that does not mean that a nearly identical approach can’t be used to boost efficiency, which is a priority in mobile processor design. So, instead of crunching countless parallelized tasks on a high-end workstation, HSA could also be used to make mobile processors more efficient and versatile.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Few people take a close look at these processors, they usually check the spec sheet when they’re buying a new phone and that’s it: They look at the numbers and brands. They usually don’t look at the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">SoC die</em> itself, which tells us a lot, and here is why: GPUs on high-end smartphone processors take up more silicon real estate than CPUs. Considering they’re already there, it would be nice to put them to good use in applications other than gaming, wouldn’t it?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A hypothetical, fully HSA-compliant smartphone processor could allow developers to tap this potential without adding much to the overall production costs, implement more features, and boost efficiency.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Here is what HSA could do for smartphone processors, in theory at least:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Improve efficiency by transferring suitable tasks to the GPU.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Boost performance by offloading the CPU in some situations.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Utilize the memory bus more effectively.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Potentially reduce chip manufacturing costs by tapping more silicon at once.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Introduce new features that could not be handled by the CPU cores in an efficient way.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Streamline development by virtue of standardisation.</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sounds nice, especially when you consider developers are unlikely to waste a lot of time on implementation. That’s the theory, but we will have to wait to see it in action, and that may take a while.</div>
<h2 id="how-does-hsa-work-anyway" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How Does HSA Work Anyway?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I already outlined the basics in the introduction, and I am hesitant to go into too much detail for a couple of reasons: Nobody likes novellas published on a tech blog, and HSA implementations can differ.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Therefore, I will try to outline the concept in a few hundred words.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On a standard system, an application would offload calculations GPU by transferring the buffers to the GPU, which would involve a CPU call prior to queuing. The CPU would then schedule the job and pass it to the GPU, which would pass it back to the CPU upon completion. Then the application would get the buffer, which would again have to be mapped by the CPU before it is ready. As you can see, this approach involves a lot of back-and-forths.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Different architectures on one memory bus. Streamlining is the gist of HSA." src="https://assets.toptal.io/uploads/blog/image/92340/toptal-blog-image-1460094958967-9184d0f7527ef0af31d22f5fb97c97ae.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Different architectures on one memory bus. Streamlining is the gist of HSA.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On an HSA system, the application would queue the job, the HSA CPU would take over, hand it off to the GPU, get it back, and get it to the application. Done.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is made possible by sharing system memory directly between the CPU and GPU, although other computing units could be involved too (DSPs for example). To accomplish this level of memory integration, HSA employs a virtual address space for compute devices. This means CPU and GPU cores <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">can access the memory on equal terms</em>, as long as they share page tables, allowing different devices to exchange data through pointers.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is obviously great for efficiency, because it is no longer necessary to allocate memory to the GPU and CPU using virtual memory for each. Thanks to unified virtual memory, both of them can access the system memory according to their needs, ensuring superior resource utilization and more flexibility.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Imagine a low-power system with 4GB of RAM, 512MB of which is allocated for the integrated GPU. This model is usually not flexible, and you can’t change the amount of GPU memory on the fly. You’re stuck with 256MB or 512MB, and that’s it. With HSA, you can do whatever the hell you want: If you offload a lot of stuff to the GPU, and need more RAM for the GPU, the system can allocate it. So, in graphics-bound applications, with a lot of hi-res assets, the system could end up allocating 1GB or more RAM to the GPU, seamlessly.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All things being equal, HSA and non-HSA systems will share the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">same memory bandwidth</em>, have access to the<em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">same amount of memory</em>, but the HSA system could end up using it much more efficiently, thus improving performance and reducing power consumption. It’s all about getting more for less.</div>
<h2 id="what-would-heterogeneous-computing-be-good-for" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
What Would Heterogeneous Computing Be Good For?</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The simple answer? Heterogeneous computing, or HSA as one if its implementations, should be a good choice for all compute tasks better suited to GPUs than CPUs. But what does <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">that</em> exactly mean, what are GPUs good at anyway?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Modern, integrated GPUs aren’t very powerful compared to discrete graphics (especially high-end gaming graphics cards and workstation solutions), but they are vastly more powerful than their predecessors.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you haven’t been keeping track, you might assume that these integrated GPUs are a joke, and for years they were just that: graphics for cheap home and office boxes. However, this started changing at the turn of the decade as integrated GPUs moved from the chipset into the CPU package and die, becoming <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">truly integrated</em>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="This is what an AMD processor die looks nowadays. We still call them processors, but the GPU takes up substantially more silicon real estate than the CPU." src="https://assets.toptal.io/uploads/blog/image/92339/toptal-blog-image-1460094932519-ed8b66cb86348167571bead85b4195c2.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
This is what an AMD processor die looks nowadays. We still call them processors, but the GPU takes up substantially more silicon real estate than the CPU.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
While still woefully underpowered compared to flagship GPUs, even integrated GPUs pack a lot of potential. Like all GPUs, they excel at single instruction, multiple data (SIMD) and single instruction, multiple threads (SIMT) loads. If you need to crunch a lot of numbers in repetitive, parallelised loads, GPUs should help. CPUs, on the other hand, are still better at heavy, branched workloads.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
That’s why CPUs have fewer cores, usually between two and eight, and the cores are optimized for sequential serial processing. GPUs tend to have dozens, hundreds, and in flagship discrete graphics cards, thousands of smaller, more efficient cores. GPU cores are designed to handle multiple tasks simultaneously, but these individual tasks are much simpler than those handled by the CPU. Why burden the CPU with such loads, if the GPU can handle them with superior efficiency and/or performance?</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
But if GPUs are so bloody good at it, why didn’t we start using them as general computing devices years ago? Well, the industry tried, but progress was slow and limited to certain niches. The concept was originally called<a href="https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">General Purpose Computing on Graphics Processing Units (GPGPU)</a>. In the old days, potential was limited, but the GPGPU concept was sound and was subsequently embraced and standardized in the form of Nvidia’s CUDA and Apple’s/Khronos Group’s OpenCL.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
CUDA and OpenCL made a huge difference since they allowed programmers to use GPUs in a different, and much more effective, way. They were, however, vendor-specific. You could use CUDA on Nvidia hardware, while OpenCL was reserved for ATI hardware (and was embraced by Apple). Microsoft’s DirectCompute API was released with DirectX 11, and allowed for a limited, vendor agnostic approach (but was limited to Windows).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s sum up by listing a few applications for GPU computing:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Traditional high-performance computing (HPC)</span> in the form of HPC clusters, supercomputers, GPU clusters for compute loads, GRID computing, load-balancing.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Loads that require physics</span>, which can, but don’t have to, involve gaming or graphics in general. They can also be used to handle fluid dynamics calculations, statistical physics, and a few exotic equations and algorithms.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Geometry</span>, almost everything related to geometry, including transparency computations, shadows, collision detection and so on.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Audio processing</span>, using a GPU in lieu of DSPs, speech processing, analogue signal processing and more.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Digital image processing</span>, is what GPUs are designed for (obviously), so they can be used to accelerate image and video post processing and decoding. If you need to decode a video stream and apply a filter, even an entry-level GPU will wipe the floor with a CPU.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Scientific computing</span>, including climate research, astrophysics, quantum mechanics, molecular modelling, and so on.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Other computationally intensive tasks</span>, namely encryption/decryption. Whether you need to “mine” cryptocurrencies, encrypt or decrypt your confidential data, crack passwords or detect viruses, the GPU can help.</div>
</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is not a complete list of potential GPU compute applications, but readers unfamiliar with the concept should get a general idea of what makes GPU compute different. I also left out obvious applications, such as gaming and professional graphics.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A comprehensive list does not exist, anyway, because GPU compute can be used for all sorts of stuff, ranging from finance and medical imaging, to database and statistics loads. You’re limited by your own imagination. So-called computer vision is another up and coming application. A capable GPU is a good thing to have if you need to “teach” a drone or driverless car to avoid trees, pedestrians, and other vehicles.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Feel free to insert your favourite Lindsay Lohan joke here.</div>
<h2 id="developing-for-hsa-time-for-some-bad-news" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Developing For HSA: Time For Some Bad News</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This may be my personal opinion rather than fact, but I am an HSA believer. I think the concept has a lot of potential, provided it is implemented properly and gains enough support among chipmakers and developers. However, progress has been painfully slow, or maybe that’s just my feeling, with a pinch of wishful thinking. I just like to see new tech in action, and I’m anything but a patient individual.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The trouble with HSA is that it’s <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">not there</em>, yet. That does not mean it won’t take off, but it might take a while. After all, we are not just talking about new software stacks; HSA requires new hardware to do its magic. The problem with this is that much of this hardware is still on the drawing board, but we’re getting there. Slowly.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Unfortunately, the HSA solution stack includes more than the standard suite of software tools. Heterogeneous computing is a symbiosis of software and hardware. " src="https://assets.toptal.io/uploads/blog/image/92341/toptal-blog-image-1460095052695-2a1028d972db5a82c85317efc53a9f2d.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
Unfortunately, the HSA solution stack includes more than the standard suite of software tools. Heterogeneous computing is a symbiosis of software and hardware.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">does not mean</em> developers aren’t working on HSA-related projects, but there’s not a lot of interest, or progress, for that matter. Here are a few resources you should check out if you want to give HSA a go:</div>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 20px; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://github.com/hsafoundation" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">HSA Foundation @ GitHub</a> is, obviously, <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">the place</em> for HSA-related resources. The HSA Foundation publishes and maintains a number of projects on GitHub, including debuggers, compilers, vital <a href="https://github.com/HSAFoundation/HSAIL-Tools" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">HSAIL tools</a>, and much more. Most resources are designed for AMD hardware.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://developer.amd.com/community/blog/2013/05/29/hsa-foundation-has-just-released-version-0-95-of-the-programmers-reference-manual-which-we-affectionately-refer-to-as-the-hsail-spec/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">HSAIL resources provided by AMD</a> allows you to get a better idea of the HSAIL spec. HSAIL stands for HSA Intermediate Language, and it’s basically the key tool for back-end compiler writers and library writers who want to target HSA devices.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://hsafoundation.app.box.com/s/m6mrsjv8b7r50kqeyyal" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">HSA Programmer’s Reference Manual</a> (PDF) includes the complete HSAIL spec, plus a comprehensive explanation of the intermediate language.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
HSA Foundation resources are limited for the time being and the foundation’s Developers Program is “coming soon,” but there are a number of official <a href="http://www.hsafoundation.com/hsa-developer-tools/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">developer tools</a> to check out. More importantly, they will give you a good idea of the stack you’ll need to get started.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The <a href="http://developer.amd.com/community/blog/tag/heterogeneous-computing/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">official AMD Blog</a> features some useful HSA content as well.</div>
</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This should be enough to get you started, provided you are the curious type. The real question is whether or not you should bother to begin with.</div>
<h2 id="the-future-of-hsa-and-gpu-computing" style="background-color: white; border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Future Of HSA And GPU Computing</h2>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Whenever we cover an emerging technology, we are confronted with the same dilemma: <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Should we tell readers to spend time and resources on it, or to keep away, taking the wait and see approach?</em></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I have already made it clear that I am somewhat biased because I like the general concept of GPU computing, but most developers can do without it, for now. Even if it takes off, HSA will have limited appeal and won’t concern most developers. However, it could be important down the road. Unfortunately for AMD, it’s unlikely to be a game-changer in the x86 processor market, but it could prove more important in ARM-based mobile processors. It may have been AMD’s idea, but companies such as Qualcomm and MediaTek are better positioned to bring HSA-enabled hardware to hundreds of millions of users.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It has to be a perfect symbiosis of software and hardware. If mobile chipmakers go crazy over HSA, it would be a big deal. A new generation of HSA chips would blur the line between CPU and GPU cores. They would share the same memory bus on equal terms, and I think companies will start marketing them differently. For example, AMD is already marketing its APUs as “compute devices” comprised of different “compute cores” (CPUs and GPUs).</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Mobile chips could end up using a similar approach. Instead of marketing a chip with eight or ten CPU cores, and such and such GPU, chipmakers could start talking about clusters, modules and units. So, a processor with four small and four big CPU cores would be a “dual-cluster” or “dual-module” processor, or a “tri-cluster” or “quad-cluster” design, if they take into account GPU cores. A lot of tech specs tend to become meaningless over time, for example, the DPI on your office printer, or megapixel count on your cheap smartphone camera.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="HSA enables different architectures to pull their own weight and tackle wildly different loads with greater efficiency. " src="https://assets.toptal.io/uploads/blog/image/92342/toptal-blog-image-1460095075231-7c4f9a5cc886888d38176b55c815c208.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background: rgb(250, 250, 250); border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; text-align: center; vertical-align: baseline; width: 864px;">
HSA enables different architectures to pull their own weight and tackle wildly different loads with greater efficiency.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It’s not just marketing though. If GPUs become as flexible as CPU cores, and capable of accessing system resources on equal terms as the CPU, why should we even bother calling them by their real name? Two decades ago, the industry stopped using dedicated mathematical coprocessors (FPUs) when they became a must-have component of every CPU. Just a couple of product cycles later, we forgot they ever existed.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Keep in mind that HSA is not the only way of tapping GPUs for computation.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Intel and Nvidia are not on board, and their approach is different. Intel has quietly ramped up GPU R&D investment in recent years, and its latest integrated graphics solutions are quite good. As on-die GPUs become more powerful and take up more silicon real estate, Intel will have to find more ingenious ways of using them for general computing.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Nvidia, on the other hand, pulled out of the integrated graphics market years ago (when it stopped producing PC chipsets), but it did try its luck in the ARM processor market with its Tegra-series processors. They weren’t a huge success, but they’re still used in some hardware, and Nvidia is focusing its efforts on embedded systems, namely automotive. In this setting, the integrated GPU pulls its own weight since it can be used for collision detection, indoor navigation, 3D mapping, and so on. Remember Google’s <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Project Tango</em>? Some of the hardware was based on Tegra chips, allowing for depth sensing and a few other neat tricks. On the opposite side of the spectrum, Nvidia’s Tesla product line covers the high-end GPU compute market, and ensures Nvidia’s dominance in this niche for years to come.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Bottom line? On paper, GPU computing is a great concept with loads of potential, but the current state of technology leaves much to be desired. HSA should go a long way towards addressing most of these problems. What’s more, it’s not supported by <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">all</em> industry players, which is bound to slow adoption further.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It may take a few years, but I am confident GPUs will eventually rise to take their rightful place in the general computing arena, even in mobile chips. The technology is almost ready, and economics will do the rest. How? Well, here’s a simple example. Intel’s current generation Atom processors feature 12 to 16 GPU Execution Units (EUs), while their predecessors had just four EUs, based on an older architecture. As integrated GPUs become bigger and more powerful, and as their die area increases, chipmakers will have no choice but to use them to improve overall performance and efficiency. Failing to do so would be bad for margins and shareholders.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Don’t worry, you’ll still be able to enjoy the occasional game on this new breed of GPU. However, even when you’re not gaming, the GPU will do a lot of stuff in the background, offloading the CPU to boost performance and efficiency.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
I think we can all agree this would be a huge deal, especially on inexpensive mobile devices.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The original article was written by<span style="color: #929292; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; text-transform: uppercase;"> </span><span style="border: 0px; box-sizing: border-box; color: #3976cb; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; font-weight: 600; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-transform: uppercase; vertical-align: baseline;"><a class="link is-blue" href="https://www.toptal.com/blog" style="border: 0px; box-sizing: border-box; color: #3863a0; display: inline; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">NERMIN HAJDARBEGOVIC</a></span><span style="color: #929292; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; text-transform: uppercase;"> - TECHNICAL EDITOR @ </span><a class="link is-blue" href="https://www.toptal.com/" style="border: 0px; box-sizing: border-box; color: #3863a0; display: inline; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; letter-spacing: 0.13px; line-height: 19px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; text-decoration: none; text-transform: uppercase; transition: color 150ms, transform, text-shadow, -webkit-transform; vertical-align: baseline;" target="_blank">TOPTAL</a> and can be read <a href="https://www.toptal.com/scientific-computing/hsa-for-developers" target="_blank">here</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: black; font-family: Arial, Helvetica, sans-serif; font-size: 13.92px; line-height: 22.272px;">If you’d like to find more resources on Toptal designers or hire a Toptal designer, check </span><a href="https://www.toptal.com/designers/resources" style="border: 0px; color: #787878; font-family: Arial, Helvetica, sans-serif; font-size: 13.92px; line-height: 22.272px; margin: 0px; outline: 0px; padding: 0px; text-decoration: none;">this</a><span style="color: black; font-family: Arial, Helvetica, sans-serif; font-size: 13.92px; line-height: 22.272px;"> out.</span></div>
Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-8706416286757464743.post-82740112450236670772016-04-07T04:13:00.001+07:002016-04-07T04:13:31.537+07:00Getting Started with Elixir Programming Language<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
If you have been reading blog posts, hacker news threads, your favorite developers tweets or listening to podcasts, at this point you’ve probably heard about the Elixir programming language. The language was created by José Valim, a well known developer in the open-source world. You may know him from the Ruby on Rails MVC framework or from the device and simple_form ruby gems him and his co-workers from the Plataformatec have been working on in the last few years.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
According the José Valim, Elixir was born in <a href="https://github.com/elixir-lang/elixir/commit/337c3f2d569a42ebd5fcab6fef18c5e012f9be5b" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">2011</a>. He had the idea to build the new language due the lack of good tools to solve the concurrency problems in the ruby world. At that time, after spending time studying concurrency and distributed focused languages, he found two languages that he liked, Erlang and Clojure which run in the JVM. He liked everything he saw in the Erlang language (Erlang VM) and he hated the things he didn’t see, like polymorphism, metaprogramming and language extendability attributes which Clojure was good at. So, Elixir was born with that in mind, to have an alternative for Clojure and a dynamic language which runs in the <a href="https://www.toptal.com/erlang" target="_blank">Erlang</a> Virtual Machine with good extendability support.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Getting Started with Elixir Programming Language" src="https://assets.toptal.io/uploads/blog/image/92321/toptal-blog-image-1459962824369-e08438fdf087f5a07565f0a162213a0b.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
<a href="http://elixir-lang.org/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Elixir</a> describes itself as a dynamic, functional language with immutable state and an actor based approach to concurrency designed for building scalable and <a href="https://github.com/h4cc/awesome-elixir" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">maintainable applications</a> with a simple, modern and tidy syntax. The language runs in the Erlang Virtual Machine, a battle proof, high-performance and distributed virtual machine known for its low latency and fault tolerance characteristics.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Before we see some code, it’s worth saying that Elixir has been <a href="https://elixirstatus.com/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">accepted by the community</a> which is growing. If you want to learn Elixir today you will easily find books, libraries, conferences, meetups, podcasts, blog posts, newsletters and all sorts of learning sources out there as well as it was accepted by the Erlang creators.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Let’s see some code!</div>
<h2 id="install-elixir" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Install Elixir:</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Installing Elixir is <a href="http://elixir-lang.org/install.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">super easy</a> in all major platforms and is an one-liner in most of them.</div>
<h4 id="arch-linux" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Arch Linux</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Elixir is available on Arch Linux through the official repositories:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">pacman -S elixir
</code></pre>
<h4 id="ubuntu" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Ubuntu</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Installing Elixir in Ubuntu is a bit tidious. But it is easy enough nonetheless.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="text-align: justify;">
<span style="background-color: #fffffc; font-size: 0.9em;">wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i erlang-solutions_1.0_all.deb</span></div>
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">apt-get update
apt-get install esl-erlang
<div style="text-align: justify;">
</div>
apt-get install elixir
</code></pre>
<h4 id="os-x" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
OS X</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Install Elixir in OS X using <a href="http://brew.sh/" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Homebrew</a>.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">brew install elixir
</code></pre>
<h2 id="meet-iex" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Meet IEx</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
After the installation is completed, it’s time to open your shell. You will spend a lot of time in your shell if you want to develop in Elixir.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Elixir’s interactive shell or IEx is a REPL - (Read Evaluate Print Loop) where you can explore Elixir. You can input expressions there and they will be evaluated giving you immediate feedback. Keep in mind that your code is truly evaluated and not compiled, so make sure not to run profiling nor benchmarks in the shell.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92322/toptal-blog-image-1459963008841-22c42ee54bd1b20ebfb318c608f8a3a2.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h3 id="the-break-command" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
The Break Command</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
There’s an important thing you need to know before you start the IEx RELP - how to exit it.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
You’re probably used to hitting <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CTRL+C</code> to close the programs running in the terminal. If you hit <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CTRL+C</code> in the IEx RELP, you will open up the Break Menu. Once in the break menu, you can hit <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">CTRL+C</code> again to quit the shell as well as pressing <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">a</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92323/toptal-blog-image-1459963049017-22b5ca5fa5de605ab203daa7069a916a.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
I’m not going to dive into the break menu functions. But, let’s see a few IEx helpers!</div>
<h3 id="helpers" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Helpers</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
IEx provides a bunch of <a href="http://elixir-lang.org/docs/master/iex/IEx.Helpers.html" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">helpers</a>, in order to list all of them type: <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">h()</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
And this is what you should see:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92324/toptal-blog-image-1459963090310-df49ceda12f72d96a48dc2a2e36c96b4.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Those are some of my favorites, I think they will be yours as well.</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">h</code> as we just saw, this function will print the helper message.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">h/1</code> which is the same function, but now it expects one argument.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
For instance, whenever you want to see the documentation of the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">String</code> <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">strip/2</code> method you can easily do:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92325/toptal-blog-image-1459963220557-fc13e6a76dd70539be7cc4a3c8ec2e08.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Probably the second most useful IEx helper you’re going to use while programming in Elixir is the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">c/2</code>, which compiles a given elixir file (or a list) and expects as a second parameter a path to write the compiled files to.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Let’s say you are working in one of the http://exercism.io/ Elixir exersices, the Anagram exercise.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
So you have implemented the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Anagram</code> module, which has the method <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">match/2</code> in the anagram.exs file. As the good <a href="https://www.toptal.com/elixir" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">developer</a> you are, you have written a few specs to make sure everything works as expected as well.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
This is how your current directory looks:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92326/toptal-blog-image-1459963271789-37af690536291590b151f475a383e8cc.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Now, in order to run your tests against the Anagram module you need to run/compile the tests.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92327/toptal-blog-image-1459963320834-393d7e11d9cd1e9711e686d9d7650fe2.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As you just saw, in order to compile a file, simply invoke the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">elixir</code> executable passing as argument path to the file you want to compile.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Now let’s say you want to run the IEx REPL with the Anagram module accessible in the session context. There are two commonly used options. The first is you can require the file by using the options <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">-r</code>, something like <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">iex -r anagram.exs</code>. The second one, you can compile right from the IEx session.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92328/toptal-blog-image-1459963364844-83c3fefd00f1706543f72a3c0fc25138.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Simple, just like that!</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Ok, what about if you want to recompile a module? Should you exit the IEx, run it again and compile the file again? Nope! If you have a good memory, you will remember that when we listed all the helpers available in the IEx RELP, we saw something about a recompile command. Let’s see how it works.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92329/toptal-blog-image-1459963423450-4f633b3a9b81bba5d5c11e0c0bd97ba0.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Notice that this time, you passed as an argument the module itself and not the file path.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As we saw, IEx has a bunch of other useful helpers that will help you learn and understand better how an Elixir program works.</div>
<h2 id="basics-of-elixir-types" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Basics of Elixir Types</h2>
<h3 id="numbers" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Numbers</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
There are two types of numbers. Arbitrary sized integers and floating points numbers.</div>
<h4 id="integers" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Integers</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Integers can be written in the decimal base, hexadecimal, octal and binary.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As in <a href="https://www.toptal.com/blog/tags/ruby" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Ruby</a>, you can use underscore to separate groups of three digits when writing large numbers. For instance you could right a hundred million like this:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">100_000_000
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Octal:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">0o444
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Hexdecimal:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">0xabc
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Binary:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">0b1011
</code></pre>
<h4 id="floats" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Floats</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Floare are IEEE 754 double precision. They have 16 digits of accuracy and a maximum exponent of around 10308.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Floats are written using a decimal point. There must be at least one digit before and after the point. You can also append a trailing exponent. For instance 1.0, 0.3141589e1, and 314159.0-e.</div>
<h3 id="atoms" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Atoms</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Atoms are constants that represent names. They are immutable values. You write an atom with a leading colon <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">:</code> and a sequence of letters, digits, underscores, and at signs <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">@</code>. You can also write them with a leading colon <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">:</code> and an arbitrary sequence of characters enclosed by quotes.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Atoms are a very powerful tool, they are used to reference erlang functions as well as keys and Elixir methods.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Here are a few valid atoms.</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">:name, :first_name, :"last name", :===, :is_it_@_question?
</code></pre>
<h3 id="booleans" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Booleans</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Of course, booleans are true and false values. But the nice thing about them is at the end of the day, they’re just atoms.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92330/toptal-blog-image-1459963479172-904a2a460cc07a3069f2434a087108a1.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h3 id="strings" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Strings</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
By default, strings in Elixir are <a href="https://en.wikipedia.org/wiki/UTF-8" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">UTF-8</a> compliant. To use them you can have an arbitrary number of characters enclosed by <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">"</code> or <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">'</code>. You can also have interpolated expressions inside the string as well as escaped characters.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92331/toptal-blog-image-1459963499300-85b682ba076110286b28fb7935233cc9.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Be aware that single quoted strings are actually a list of binaries.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92332/toptal-blog-image-1459963556457-a80650d8ff99438a60611a914bb01c05.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h3 id="anonymous-functions" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Anonymous Functions</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
As a functional language, Elixir has anonymous functions as a basic type. A simple way to write a function is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">fn (argument_list) -> body end</code>. But a function can have multiple bodies with multiple argument lists, guard clauses, and so on.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Dave Thomas, in the <a href="https://pragprog.com/book/elixir/programming-elixir" rel="noopener noreferrer" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Programming Elixir</a> book, suggests we think of fn…end as being the quotes that surround a string literal, where instead of returning a string value we are returning a function.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92333/toptal-blog-image-1459963581791-8677d3008dfa070cd66c8fd75fd7b0fb.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h3 id="tuples" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Tuples</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Tuple is an immutable indexed array. They are fast to return its size and slow to append new values due its immutable nature. When updating a tuple, you are actually creating a whole new copy of the tuple self.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Tuples are very often used as the return value of an array. While coding in Elixir you will very often see this, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">{:ok, something_else_here}</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Here’s how we write a tuple: <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">{?a,?b,?c}</code>.</div>
<h3 id="pattern-matching" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Pattern Matching</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
I won’t be able to explain everything you need to know about Pattern Matching, however what you are about to read covers a lot of what you need to know to get started.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Elixir uses <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">=</code> as a match operator. To understand this, we kind of need to unlearn what we know about <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">=</code> in other traditional languages. In traditional languages the equals operator is for assignment. In Elixir, the equals operators is for pattern matching.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
So, that’s the way it works values in the left hand side. If they are variables they are bound to the right hand side, if they are not variables elixir tries to match them with the right hand side.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92334/toptal-blog-image-1459963608317-f424527db966933563936ead380eb82e.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h4 id="pin-operator" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Pin Operator</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Elixir provides a way to always force pattern matching against the variable in the left hand side, the pin operator.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92335/toptal-blog-image-1459963637820-6b65dac181da0df0c695cfac810e6e8d.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h3 id="lists" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Lists</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
In Elixir, Lists look like arrays as we know it from other languages but they are not. Lists are linked structures which consist of a head and a tail.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92336/toptal-blog-image-1459963697910-feb866e88cbeffaafb4765112aa6c850.gif" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;" /></div>
<h4 id="keyword-lists" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Keyword Lists</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Keyword Lists are a list of Tuple pairs.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
You simply write them as lists. For instance: [{:one, 1}, 2, {:three, 3}]. There’s a shortcut for defining lists, here’s how it looks: [one: 1, three: 3].</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
In order to retrieve an item from a keyword list you can either use:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">Keyword.get([{:one, 1}, 2, {:three, 3}], :one)
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Or use the shortcut:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">[{:one, 1}, 2, {:three, 3}][:one]
</code></pre>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Because keyword lists are slow when retrieving a value, in it is an expensive operation, so if you are storing data that needs fast access you should use a Map.</div>
<h4 id="maps" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Maps</h4>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Maps are an efficient collection of key/value pairs. The key can have any value you want as a key, but usually should be the same type. Different from keyword lists, Maps allow only one entry for a given key. They are efficient as they grow and they can be used in the Elixir pattern matching in general use maps when you need an associative array.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Here’s how you can write a Map:</div>
<pre style="border: 0px; box-sizing: border-box; color: #303030; font-size: 15px; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;"><code class="language-elixir" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">%{ :one => 1, :two => 2, 3 => 3, "four" => 4, [] => %{}, {} => [k: :v]}
</code></pre>
<h2 id="conclusion" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Conclusion</h2>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
Elixir is awesome, easy to understand, has simple but powerful types and very useful tooling around it which will help you when beginning to learn. In this first part, we have covered the various data types Elixir programs are built on and the operators that power them. In later parts we will dive deeper into the world of Elixir - functional and concurrent programming.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: justify; vertical-align: baseline;">
This post originally appeared in <a href="https://www.toptal.com/elixir/getting-started-elixir-programming-language" target="_blank">Toptal</a> </div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8706416286757464743.post-57480951297767659862016-03-28T23:08:00.001+07:002016-03-28T23:08:08.016+07:00The Vital Guide to Web Design Interviewing<h3 id="the-challenge" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The Challenge</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Hiring <a href="https://www.toptal.com/designers/" target="_blank">web designers</a> can be difficult. There are many disciplines under the common term ‘designer,’ and experts in one field may be novices in another, while others are a “Jack of all trades, master of none.” To make things worse, people commonly relate “design” with subjective decisions, personal preference and individual style.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All that makes the search for a great web designer seem like a daunting task. It doesn’t have to be like that; the goal of this guide is to help you find the perfect match for your team or project. This can be achieved by better understanding the different roles a web designer fills, and by having a good idea of the sort of strategic questions you can ask during the hiring process.</div>
<h3 id="web-design-disciplines-and-roles" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Web design disciplines and roles</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The “web design” profession encompasses many skills, and sometimes, separate professions. A Web Designer nowadays has to wear many hats and may have varying levels of proficiency and experience in different fields.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="A Web Designer nowadays has to wear many hats and may have varying levels of proficiency and experience in different fields." src="https://assets.toptal.io/uploads/blog/image/91746/toptal-blog-image-1444375577550-1e4c0e7622c82bcb4b176766f58615a9.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background-color: #fafafa; border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; vertical-align: baseline; width: 868px;">
A Web Designer nowadays has to wear many hats and may have varying levels of proficiency and experience in different fields.</div>
<div class="tweet_this" style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: -10px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<iframe allowtransparency="true" class="twitter-share-button twitter-share-button-rendered twitter-tweet-button" data-url="https://www.toptal.com/designers/web#hiring-guide" frameborder="0" id="twitter-widget-2" scrolling="no" src="https://platform.twitter.com/widgets/tweet_button.64a917b4a230f163048902c783e1530f.en.html#dnt=true&id=twitter-widget-2&lang=en&original_referer=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&size=m&text=A%20Web%20Designer%20nowadays%20has%20to%20wear%20many%20hats%20and%20may%20have%20varying%20levels%20of%20proficiency%20and%20experience%20in%20different%20fields.&time=1459180850643&type=share&url=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&via=toptalllc" style="border-width: 0px; box-sizing: border-box; height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: static; vertical-align: baseline; visibility: visible; width: 93px !important;" title="Twitter Tweet Button"></iframe></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It’s important for you to know some of the broader terms so you can better understand what kind of design professional is best suited for the role you are trying to fill. This will also help you clarify your project description and weed out some candidates before you proceed to the actual interview stage.</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Visual Design</span> – Skills used during the final stages of design, the visual design phase. Generally found in designers with a more artistic profile, not necessarily well versed in the technical side of web design (although many are). These, however, are nice additional skills you can look for in a web designer on top of their technical skills. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: visual design, mood boards, Illustrations, banners, photo manipulation and compositions</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Branding and logo design</span> – A profession on its own, branding and logo design are also skills that many web designers have at a rudimentary level. These are nice to have as extras, if that’s part of the project requirement or the job description of the open position you have. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: Style guides, brand books, color schemes</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">User-Experience Design (UX)</span> - Going hand in hand with IA and IxD, UX is the broad discipline of ensuring digital products work, based upon users’ expectations, providing the fastest, most painless workflow, while achieving the goals of the product. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: User personas, workflow charts, low fidelity sketches, accessibility analysis, usability tests, wireframes</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">User-Interface Design (UI)</span> – This is the practice of creating individual control elements and design of broader systems and visual language that makes the usage of a website or application nice and easy. If you are designing applications (mobile, web or otherwise), you will need someone with good UI skills in your team. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: High fidelity sketches, working prototypes, pattern libraries, UI kits</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Information Architecture (IA)</span> – The art and science of defining the optimal content structure and the clearest navigation methods. For larger websites and applications with a lot of content and different content types, or complex content structure, it’s important to have someone with experience in IA on the team. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: site maps, navigation lists, taxonomies, content audit, user journeys</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Interaction Design (IxD)</span> – Everything that deals with the interaction between human and machine generally falls into web design. With websites more and more having app-like diverse functionality, and with the myriad of interactive elements users have become used to, it’s nice to work with a web designer who is well versed in good and bad practices of IxD, who understands the well established conventions and knows when to break the rules to achieve a specific goal. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: interaction and functionality libraries, interactive prototypes, workflow charts, state maps</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Front-End Development</span> – This is the part of development that involves code employed to render the user-facing side of a website or application, and to handle interactions between user and machine on a technical level. The most technical of the design-related disciplines, and often considered a profession on its own, front-end development mainly involves good skills in HTML, CSS, and JavaScript. Good front-end developers also utilise various assistive tools such as CSS preprocessors (LESS and SASS), task runners (Gulp or Grunt), and others like npm, Bower, or Yeoman. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: Production level HTML, CSS and JavaScript code, tools for handling design changes and sometimes environment migration</em></div>
</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Note that this distinction is sometimes artificial, and although there are highly specialised experts who excel at just one of these disciplines, it’s very important for them to have a good understanding of at least a few of the others. So, even if a designer’s only job is to provide the best possible wireframes for a website, s/he must also understand how that wireframes will be turned into a working, responsive set of HTML and CSS files, and how much JS will be needed to implement the intended interactions.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Many of the best freelance web designers have solid experience in a majority of design disciplines. They are used to being a one-person show (or rather, a one-person army), handling web design projects from the initial specification talks to the final production code and everything in between. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Web professionals of this profile are great for small-sized and even some medium-sized projects with limited budget and tight deadlines.</em> They are efficient and know how to achieve 80 percent of the results with 20 percent of the effort.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Larger projects benefit more from an in-depth look at each component of the design, and require a design team comprised of highly specialised experts in their own fields. Depending on the specifics and the complexity of the project, multiple roles could be filled by a single professional or by separate specialised teams for IA, IxD, visual design, and so on.</div>
<h3 id="web-design-workflow" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Web Design Workflow</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Web design as a profession has evolved a lot over the last 10 years. Effective workflows and practices have emerged and have proved to be the de-facto industry standard. However, there are still certain practices, remnants of the early years of the web design, which should be avoided.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
One such ineffective and outdated practice is the “three mockups” approach. In the past, companies that have needed web design services have asked designers to provide three (usually) Photoshop mockups (or other forms of high-fidelity comps) to choose from. These are usually based on a set of initial brief requirements or a couple of talks with the client. The final product of this approach is design-based on personal preferences and subjective choices. Chasing user needs and achieving business goals this way is like shooting in the dark. Working this way (and requesting it from a web designer) should be avoided.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A much better approach to web design is the iterative process introduced by <a href="http://www.jjg.net/elements/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Jesse James Garrett in The Elements of User Design</a>. It involves five stages, each based on decisions made, and work done, in the previous step.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Strategy, Scope, Structure, Skeleton, Style: Meet the five stages of web design." src="https://assets.toptal.io/uploads/blog/image/91745/toptal-blog-image-1444375557660-c9058d8043e0d59fac1a46161f715448.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background-color: #fafafa; border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; vertical-align: baseline; width: 868px;">
Strategy, Scope, Structure, Skeleton, Style: Meet the five stages of web design.</div>
<div class="tweet_this" style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: -10px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<iframe allowtransparency="true" class="twitter-share-button twitter-share-button-rendered twitter-tweet-button" data-url="https://www.toptal.com/designers/web#hiring-guide" frameborder="0" id="twitter-widget-3" scrolling="no" src="https://platform.twitter.com/widgets/tweet_button.64a917b4a230f163048902c783e1530f.en.html#dnt=true&id=twitter-widget-3&lang=en&original_referer=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&size=m&text=Strategy%2C%20Scope%2C%20Structure%2C%20Skeleton%2C%20Style%3A%20Meet%20the%20five%20stages%20of%20web%20design.&time=1459180850644&type=share&url=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&via=toptalllc" style="border-width: 0px; box-sizing: border-box; height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: static; vertical-align: baseline; visibility: visible; width: 93px !important;" title="Twitter Tweet Button"></iframe></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Included here we have a very condensed version of the work involved in each stage:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Strategy</span> – Defining the key business goals of the product and balancing them with the user needs of the target audience (based on market research, focus groups, user personas, and the like). <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: High level brief, design team requirements, project objectives</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Scope</span> – Documenting the required functionality and the needed content. Also involves deciding what is to be built, and what isn’t, as part of the current project. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: Detailed project specification</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Structure</span> – Information architecture and interaction design. At this stage, the structure of the website, and its pages, is decided via card-sorting and user-journey maps. For applications, workflow charts and state maps are created. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: site map, low fidelity prototypes or wireframes</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Skeleton</span> – UI design, information design and navigation. With the structure in place, objective decisions can be made about laying out content, what UI elements to use and how they would work. All navigation elements should be implemented at this point and content added to its proper places. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Deliverables: fully functional prototype of the website or application</em></div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Style</span> – Applying the visual treatment and the brand’s style guide to the working product. With a completely functional and properly laid out website, it’s much easier to apply corporate or product branding and make objective choices about its visual treatment.</div>
</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is an iterative process and each step can go through several cycles until it’s approved. During each step, it’s also possible to find flaws, or ways to improve the previous, and change the previous set of deliverables to reflect that. The main advantage of the ability to run usability tests at each step is avoiding large commitments of time and budget on ideas which later would prove to be fundamentally wrong or suboptimal.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Web designers well versed in modern practices and workflows should be acquainted with good tools for prototyping and wireframing such as UXpin, Balsamiq or Axure. While some may prefer creating the wireframes in Photoshop, Fireworks or InDesign, others implement them straight into popular CSS frameworks like Bootstrap or Foundation. The advantage of the latter is that these early prototypes later evolve into actual production templates. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">This eliminates dead-end deliverables and reduces production time.</em></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Finally, the approach described here, as well as similar methodologies, lead to much better informed, researched and data-driven choices and use fewer subjective decisions throughout the whole design process. As such, you can easily identify designers who practice this by asking them about the reasoning behind different elements’ layout, position and style of a project they worked on. <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">They should be able to give you swift and concise answers backup up by facts or research results.</em></div>
<h3 id="responsive-web-design-process" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Responsive Web Design Process</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
These days, with a large percentage of internet traffic coming from mobile devices of different capabilities, it’s crucial that any new website be usable on as many devices as possible. The process of designing and developing websites suited for myriad screens and devices is commonly referred to as “Responsive Web Design” or sometimes as “Adaptive Design” (which is also used to refer to a specific methodology in responsive design).</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Today the question whether to have a responsive website is no longer relevant; the answer is a clear “Yes!” and even Google is placing websites not suited for mobile devices behind those that are. The real question is how to execute a successful and effective multi-device strategy without going over budget or missing the point of mobile user experience.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Any experienced web designer should be well versed in the techniques that make a website responsive. To help you find the best matches for your project, we have prepared a few questions and guidelines. There are some important considerations that must be addressed when designing and developing a responsive website.</div>
<h3 id="content-strategy-across-devices" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Content Strategy Across Devices</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Responsive design is not just about fitting all your content into any screen size; the designer has to take into consideration the context in which each device would be used along with its capabilities.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="Thanks to the mobile revolution, web designers have to take into account numerous different software and hardware platforms." src="https://assets.toptal.io/uploads/blog/image/91747/toptal-blog-image-1444375654993-5697c14fe443347e98380ce24a01bdc5.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div class="pop_out_box is-full_width is-big" style="background-color: #fafafa; border: 0px; box-sizing: border-box; color: #505050; float: none; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.5em; margin: 0px 30px 15px 0px; min-height: 0px; min-width: 0px; overflow: hidden; padding: 1em 1.5em; vertical-align: baseline; width: 868px;">
Thanks to the mobile revolution, web designers have to take into account numerous different software and hardware platforms.</div>
<div class="tweet_this" style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: -10px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<iframe allowtransparency="true" class="twitter-share-button twitter-share-button-rendered twitter-tweet-button" data-url="https://www.toptal.com/designers/web#hiring-guide" frameborder="0" id="twitter-widget-4" scrolling="no" src="https://platform.twitter.com/widgets/tweet_button.64a917b4a230f163048902c783e1530f.en.html#dnt=true&id=twitter-widget-4&lang=en&original_referer=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&size=m&text=Thanks%20to%20the%20mobile%20revolution%2C%20web%20designers%20have%20to%20take%20into%20account%20numerous%20different%20software%20and%20hardware%20platforms.&time=1459180850646&type=share&url=https%3A%2F%2Fwww.toptal.com%2Fdesigners%2Fweb%23hiring-guide&via=toptalllc" style="border-width: 0px; box-sizing: border-box; height: 20px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; position: static; vertical-align: baseline; visibility: visible; width: 93px !important;" title="Twitter Tweet Button"></iframe></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sometimes, it’s better to skip certain pieces of copy on a mobile device, use alternative copy or different image assets because the navigation of a website needs to change between different screens. Other times, specific pieces of content or functionality should be enabled only on mobile devices, such as a “click to call us” button, offer interactions based on a user’s location or show an “app download” button for the specific devices being used.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: Do we have to prepare different content or assets for different devices?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sometimes, yes. Here are instances where such changes are required:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Most often images have to be cropped differently for a small portrait screen; a wide aspect ratio image is great for a desktop website banner but is almost unusable on an upright smartphone screen.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Where a large presentational video might be great on a computer screen, it could be replaced with an image and text on mobile devices, especially if you expect traffic from devices with slower mobile internet connection.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Some pieces of copy may have to be omitted (or rewritten) for small screen devices where the user likely won’t read it.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Certain call to action controls might be changed to better suit the device; for example, “Send a Message” on computers might be replaced by “Call Now” on mobile phones.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">On small screens, complex graphs, charts and long tables are better left as linked stand-alone pages rather than have them in the content stream of the page. Another idea is to present the same data in a different way, or only show the most important parts.</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Navigation should be rethought, or even designed separately, for different screen sizes. This is not necessarily about its visualisation, but often includes different structure, such as a flat list of links instead of dropdown/drill-down menus, or by showing less levels of depth in more complex menus on specific devices.</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Layout Optimisation on Different Screens</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
While computers and larger tablets held sideways offer a lot of horizontal space to lay out the website’s content, the smaller the screen gets the less space you have for elements placed side by side. That’s why websites on smartphones are usually designed to have a single column of layout. This is one of the primary concerns when designing a responsive website: When and how should the page layout change.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: How do you make sure content layout looks good across different devices? What technology do you use to achieve that?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A few “break points” should be defined based on popular device sizes, types and context. These are predefined screen widths (and less frequently screen heights) where the page layout changes, for example, from three columns, to two and then to a single column. Currently, the most popular width breakpoints are:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1920 and up: TV screens and large desktop monitors</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1280 to 1920: for the huge majority of laptops, many modern desktop monitors as well as large tablets (usually 10” and up) when held in landscape mode (held horizontally)</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">800 to 1280: for smaller tablets in landscape mode as well as older or smaller monitors</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">480 to 800: for tablets in portrait mode (held vertically) and smartphones in landscape mode</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">up to 480: smartphones in portrait mode</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
To use different styles based on screen size <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">@media</code> is used in CSS code, for instance, to change a paragraphs’ font size to 14 pixels only on devices with a screen width larger than 480 pixels but smaller than 800, the following rule is used:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">css @media (min-width: 480px) and (max-width: 799px) { font-size: 14px; }</code></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another important consideration is to place HTML code in the same order in which it needs to be displayed on mobile devices. In general, writing clean, well-structured and semantically correct HTML code goes a long way towards the smooth implementation of a responsive website.</div>
<h3 id="interaction-with-the-user-interface-on-different-devices" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Interaction with the user interface on different devices</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since devices differ not only in screen size but also in terms of input methods, make sure every UI element works as expected in the context of each device type. This means that dropdown menus should be acceptable for computer screens but on smartphones and tablets, users would expect navigation methods more akin to those on mobile apps.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: What do you do to make sure the UI works well and feels natural on different devices? Name a couple of interaction patterns which are not suited for specific devices.</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Different devices come with different capabilities, and users expect websites on their device to function in similar fashion to the apps on their device.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The most important distinction between a desktop and a mobile UI is that desktops are usually controlled with a mouse or trackpad and a fast and easy to use keyboard, while mobile devices rely on a touch screen with no pointer and with an on-screen keyboard that’s often a hassle to use. Another consideration is that devices with no pointer also lack the hover state which is frequently used to trigger certain actions on web pages. The two input methods, pointer and touchscreen, also make different actions easier (more natural) or more difficult and slower. For example, moving items across the screen or dragging is easier on touch screens (therefore, dragging is avoided on desktops), while clicking on smaller controls is much easier with a mouse pointer (thus UI controls should be made larger on touch screens).</div>
<h3 id="asset-optimisation-based-on-screen-size" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Asset Optimisation Based on Screen Size</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Even when the same assets can be served to different devices it doesn’t mean the same image size or video quality will be optimal. To reduce load times, especially on mobile internet connection, web designers should be aware of the assets that they serve to different devices.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For example, a 1920 pixel wide background photo of 400 kilobytes, which is fine for desktop computers, will be overkill (quality wise) and slow to download (file size wise) on a smartphone. So, it’s good to have a smaller version of the image, which would be served if the user’s screen is small enough. And you don’t want the user to download both versions while s/he only sees one.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: Does it matter if you serve the same assets regardless of screen size? Is there any difference between images and backgrounds in this regard?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It’s important, at least for larger image files, to have separate versions for mobile devices and desktop screens. Smaller copies of the same image can be served to mobile users to reduce load times. Image size, however, shouldn’t be greatly reduced (if feasible) for mobile devices since they usually have screen density that’s much higher than desktop monitors. Decisions should be made on a case-by-case basis since some images can be reduced in size without much visual impact while for others it’s important to retain their details.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On a technical level, there’s a big difference between background images (which are defined in CSS) and content images (included as image files in HTML). Backgrounds can easily be set separately in different media queries in CSS, so each version is only served if the user screen matches a certain query; the others are not downloaded from the web server. For images in HTML, there is still no built-in, working and well-supported way to serve different files based on user screen size. Different techniques can be used to achieve that, such as by using polyfill scripts that simulate the behaviour of the upcoming <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><picture></code> elements, or other scripts that have conventions of their own, or using CSS background to show an image.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Generally, the CSS method, however, <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">should be avoided</em> because a CSS image background doesn’t have any semantic meaning and is considered decoration. Further, it’s lacking in accessibility as it can’t be described with the <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">title</em> and <em style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">alt</em> attributes that an <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><img></code> tag has.</div>
<h3 id="seo-semantics-content-syndication-and-accessibility-concerns" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
SEO, semantics, content syndication and accessibility concerns</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
We’re now living in a highly connected digital world where your website content is not read only by people on screens but also by machines. Search engines crawl your website to find out what it’s all about and help users by showing them your content when they look for it; people use apps to aggregate content to read at their leisure; disabled users rely on machine assistance to access and interact with your content.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All of this is good both for your website and for its audience. To make sure all of the above is possible, and done correctly, your website should adhere to certain standards and conventions. The closest it sticks to them, the better chance you have for machines to interpret it correctly.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: How do you make sure a website is well suited for SEO and the content is machine readable?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The first and most important step to making a website machine readable and SEO ready is to write semantically correct HTML markup code, using the new HTML5 elements to mark each piece of content properly.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: Are you concerned with accessibility and what do you do to improve a website in that regard?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Properly marked up, semantically correct content, alone, gives a huge boost to accessibility. Making the website accessible to the widest possible audience of users with disabilities is a valid concern in a project; there are additional steps that should be taken to improve on that regard:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Aria roles</span> – in addition to the semantic HTML5 tags, aria roles can be assigned to the more important elements of content in order to show in greater detail each one’s purpose. This helps assistive tools understand the content, thus making it easier to use.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Color combination concerns</span> – many users have different types and degrees of color blindness, so in UI design, it’s important to avoid combining certain colors. It’s good practice to not rely on colors alone as a distinction between two UI elements (e.g. unlabeled green and red buttons or indicators).</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span style="border: 0px; box-sizing: border-box; font-weight: 600; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">User control</span> – The designer must also make sure the user has some degree of control over the way the website looks. Most importantly, the browser’s zoom function must not make the site unusable. Additionally, the content on a correctly marked up page is easily picked by reading apps and re-styled, as per the user’s preference (larger font sizes, a more contrasting color scheme, a more readable typeface, for example).</div>
</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: What about content syndication? What steps should we take to make sure our content can be distributed over different channels and used by other apps?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There are several approaches that can be taken to assure the content can be interpreted outside the context of the website. Choosing the right format depends on the content itself, and the intended syndication channels, but there are a few common things to consider:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Semantic HTML5 markup is usually the most important and common way to make content redistributable. Make sure the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><article></code> tag is used to mark up individual pieces of content, each one with its own heading. Article headers and footnotes can be marked up with the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><header></code> and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><footer></code> tags respectively, and related content can be marked up as <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><aside></code>. Dates should be properly marked as <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><time datetime=“yyyy-mm-dd hh:mm:ss"></code> and addresses as <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;"><address></code>, and so on.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you need clearly defined pieces of content, with their unique characteristics specified, consider using Schema.org to make sure everything is properly listed according to its widely accepted schemas.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
On a technical level, some of the formats that can be used to define specific data structure are RDF, Microformats and Microdata. They use HTML compatible elements (classes or tag attributes) to simulate XML-like structured data.</div>
</li>
</ul>
<h3 id="content-management-systems-premium-themes-css-frameworks" style="border: 0px; box-sizing: border-box; color: #3863a0; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.3em; line-height: 1.3em; margin: 2em 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Content Management Systems, Premium Themes, CSS Frameworks</h3>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
As the web has matured, developers have created an abundance of well-made tools and apps to assist the job of a web designer, or developer, while adhering to well-established workflows and design patterns.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Content Management Systems (CMS) allow site owners and admin control over the website without requiring them to dabble in code, yet leaving freedom for the designers and developers who use them.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: In your opinion, what are the advantages and disadvantages of using a CMS? When do you think it’s good to use one, and what are the alternatives?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Although many web designers have strong preference over one CMS or another, they should be able to objectively decide whether to use one.</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Advantages of CMS: ease of content management, built-in template engines for smoother development, adhering to best development practices, well documented code that can easily be handed to another developer, many available plugins that save hundreds of hours of development work and are well supported, to name a few.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Disadvantages of CMS: when used incorrectly, may lead to bloated code and slower load times, the tendency to use heavy plugins for small features easily implemented by a few lines of code, limitations for more complex data structures or interactions where the developer may have to “fight” the system to achieve goals.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
It’s a good idea to use a popular CMS for almost any website project where the client needs to have full control over the content. Some larger sites, and especially web apps however, are too complex for conventional CMS, and will perform far better if implemented using a custom made system, tailored to specifics and needs. In many cases, a good CMS can handle these types of projects as well; however, that comes at the cost of heavy modifications, changing fundamental concepts, losing compatibility with plugins and the ability to update or receive support.</div>
</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The so called “Premium Themes” for popular CMS are a common choice for projects with limited budget or tight deadlines. The built-in features and layout, however, come at a price. Good web designers understand that and can help you decide if this is the right approach for your project, can evaluate the usability of a theme and advise you about its pros and cons.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<span class="guide-question" style="border: 0px; box-sizing: border-box; color: #2557a1; font-style: italic; font-weight: bold; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Q: What are some of the common issues that we face when using off-the-shelf “Premium” themes for a CMS?</span></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Despite the seemingly easily customised components and styles, most of the themes they provide usually come at a cost (performance wise) and have significant limitations. A few common problems found in the popular themes are:</div>
<ul style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Loading and executing a lot more code and assets than needed. Theme developers strive to outclass one another by including more and more features and capabilities in their themes. That, combined with the fact that themes are made to be flexible without editing any programming code, leads to a bloated code base with too many scripts and the need to query and process more data than standard templates.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Themes’ looks and layouts are hard to modify beyond their intended customisation capabilities. Again, due to a code base significantly more complex compared to the standard CMS templates, heavy modifications take much longer and there’s always a chance those changes could break something else.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If the project requires the use of complex plugins, which add templates of their own, styling those to match the rest of the theme’s visual style is also much harder.</div>
</li>
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><div style="border: 0px; box-sizing: border-box; font-size: 1em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Finally, despite all their customisation options, themes tend to guide the website’s design direction, placing it on ‘rails’ and limiting the creative potential of the designer. This leads to a form over function approach, where content is made for the layout instead the other way around.</div>
</li>
</ul>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Another way to reduce development times and budget is by using CSS frameworks such as Twitter’s Bootstrap or ZURB’s Foundation. These frameworks come loaded with reusable pieces of code, markup conventions, commonly used javascript interactions and built-in best practices. They also use development tools such as CSS preprocessors (LESS or SASS), task runners (Gulp or Grunt), code linters and minifiers to improve the quality and size of the production code.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can check our <a href="https://www.toptal.com/designers/web/interview-questions" target="_blank">Web Designer Interview Questions</a> article for a sample question about frameworks and front-end development tools.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Of course, all these questions and notes are barely scratch the surface of web design and development practices.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A good designer should be able to give you answers similar to those suggested here, but they may also have other ideas about how to turn your project into a success. Good communication is key. Exceptional web designers also have the ability to understand requirements along with the underlying reasoning behind them. That’s why they are able to come up with better ways to solve problems while also providing solutions for problems hidden beneath the surface. In the end, the skill which unifies all design disciplines is the ability to solve problems.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<i>This hiring guide originally appeared in <a href="https://www.toptal.com/designers/web#hiring-guide" target="_blank">Toptal Engineering blog</a></i></div>
Anonymousnoreply@blogger.com10tag:blogger.com,1999:blog-8706416286757464743.post-20168136169525147982016-03-23T02:57:00.001+07:002016-03-23T02:57:38.853+07:00To Python 3 and Back Again: Is It Worth the Switch?<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<a href="https://www.toptal.com/python" target="_blank">Python</a> 3 has been in existence for 7 years now, yet some still prefer to use Python 2 instead of the newer version. This is a problem especially for neophytes that are approaching Python for the first time. I realized this at my previous workplace with colleagues in the exact same situation. Not only were they unaware of the differences between the two versions, they were not even aware of the version that they had installed.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Inevitably, different colleagues had installed different versions of the interpreter. That was a recipe for disaster if they would’ve then tried to blindly share the scripts between them.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This wasn’t quite their fault, on the contrary. A greater effort for documenting and raising awareness is needed to dispel that veil of FUD (fear, uncertainty and doubt) that sometimes affects our choices. This post is thus thought for them, or for those who already use Python 2 but aren’t sure about moving to the next version, maybe because they tried version 3 only at the beginning when it was less refined and support for libraries was worse.</div>
Two Dialects, One Language<br />
A Concrete Example<br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># encoding: utf-8</span></code><code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">
</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">from</span> os <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> listdir, stat</code><code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">
</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># to keep this example simple, we won't use the `pwd` module</span></code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">names = {<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1000</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'dario'</span>,</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1001</span>: <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">u'олга'</span>}</code><code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">
</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> node <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span> listdir(<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">b'.'</span>):</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> owner = names[stat(node).st_uid]</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> print(owner + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">': '</span> + node)</code><br />
<code class="language-bash hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">su олга -c <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">"touch é"</span></code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">`TypeError: Can't convert 'bytes' object to str implicitly`</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">print(owner + <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">': '</span> + node)</code><br />
Tools for Automated Conversion<br />
Python 3 Is Not Just about Unicode<br />
Optional Keyword Arguments<br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">merge_dicts({<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'a'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'c'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>}, {<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'a'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">4</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'b'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>}, {<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'b'</span>: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">-1</span>})</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># {'b': -1, 'a': 4, 'c': 3}</span></code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">from</span> operator <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">import</span> add</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">merge_dicts({<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'a'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">1</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'c'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">3</span>}, {<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'a'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">4</span>, <span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'b'</span>:<span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">2</span>}, {<span class="hljs-string" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">'b'</span>: <span class="hljs-number" style="border: 0px; box-sizing: border-box; color: #2aa198; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">-1</span>}, withf=add)</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-comment" style="border: 0px; box-sizing: border-box; color: #93a1a1; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"># {'b': 1, 'a': 5, 'c': 3}</span></code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">second</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(a, b)</span>:</span></code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> b</code><code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">
</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"><span class="hljs-function" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">def</span> <span class="hljs-title" style="border: 0px; box-sizing: border-box; color: #268bd2; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">merge_dicts</span><span class="hljs-params" style="border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">(*dicts, withf=second)</span>:</span></code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> newdict = {}</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> d <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span> dicts:</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> shared_keys = newdict.keys() & d.keys()</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> newdict.update({k: d[k] <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> k <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span> d.keys() - newdict.keys()})</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> newdict.update({k: withf(newdict[k], d[k]) <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">for</span> k <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">in</span> shared_keys})</code><br />
<code class="language-py hljs" style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #657b83; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> <span class="hljs-keyword" style="border: 0px; box-sizing: border-box; color: #859900; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">return</span> newdict</code><br />
Unpacking Operator<br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">a, b, *rest = [1, 2, 3, 4, 5]</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">rest</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"># [3, 4, 5]</code><br />
Simpler APIs for Iterables<br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">zip(itertools.count(1), 'abc')</code><br />
Function Annotations<br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">@get('/balance')</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">def balance(user_id: int):</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> pass</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> </code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">from decimal import Decimal</code><code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">
</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">@post('/pay')</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;">def pay(user_id: int, amount: Decimal):</code><br />
<code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.9em; line-height: 1.5em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; overflow-x: auto; padding: 2px 5px; vertical-align: text-bottom; width: 864px;"> pass</code><br />
Wrapping Up<br />
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Several functions/constructors now return <a href="https://www.python.org/dev/peps/pep-0343/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">contextmanagers</a>, simplifying closing up objects and managing errors: <a href="https://docs.python.org/3/library/gzip.html#gzip.open" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">gzip.open</a> (already from 2.7), <a href="https://docs.python.org/3/library/mmap.html#mmap.mmap" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">mmap</a>, <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ThreadPoolExecutor</a>, <a href="https://docs.python.org/3/library/stdtypes.html#memoryview" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">memoryview</a>, <a href="https://docs.python.org/3/library/ftplib.html#ftplib.FTP" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">FTP</a>, <a href="https://docs.python.org/3/library/tarfile.html#tarfile.TarFile" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">TarFile</a>, <a href="https://docs.python.org/3/library/socket.html#socket.create_connection" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">socket.create_connection</a>, <a href="https://docs.python.org/3/library/select.html#select.epoll" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">epoll</a>, <a href="https://docs.python.org/3/library/nntplib.html#nntplib.NNTP" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">NNTP</a>, <a href="https://docs.python.org/3/library/smtplib.html#smtplib.SMTP" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">SMTP</a>, <a href="https://docs.python.org/3/library/aifc.html#aifc.open" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">aifc.open</a>, <a href="https://docs.python.org/3/library/shelve.html#shelve.Shelf" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">Shelf</a>, and more.</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/lzma.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">lzma</a>, for an improved compression compared to gzip</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/asyncio.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">asyncio</a>, for asynchronous programming in Python</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/pathlib.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">pathlib</a>, a more pythonic and expressive way to manipulate paths</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/ipaddress.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">ipaddress</a></li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/functools.html#functools.lru_cache" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">lru_cache</a>, to automatically cache the results of expensive functions</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">mock (the same module mentioned above, previously available only from PyPI)</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">A more efficient <a href="http://www.python.org/dev/peps/pep-0393" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">string representation</a> in memory</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;"><a href="https://docs.python.org/3/library/collections.html#ordereddict-objects" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">OrderedDict</a>, everyone needed it at least once</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">autocompletion inside pdb</li>
</ul>
<ul style="border: 0px; box-sizing: border-box; font-size: 1.2em; list-style: none; margin: 0px 0px 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; line-height: 1.5em; list-style-type: disc; margin: 0px 0px 0.75em 30px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">the __pycache__ directory, that helps to avoid littering every other project folder with <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.pyc</code> files</li>
</ul>
<br />
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
First of all, is it true that Python 2 and Python 3 are different languages? This is not a trivial question. Even if some people would settle the question with: <a href="https://mail.python.org/pipermail/python-3000/2006-July/002552.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">“No, it’s not a new language”</a>, as a matter of fact <a href="https://www.python.org/dev/peps/pep-3099/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">several proposals that would have broken compatibility without yielding important advantages have been rejected</a>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="To Python 3 and Back Again: Is It worth the Switch?" src="https://assets.toptal.io/uploads/blog/image/92216/toptal-blog-image-1457618659472-be2f380fe3aad41333427ecd5a1ec5c5.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Python 3 is a new version of Python, but it’s not necessarily backwards compatible with code written for Python 2. At the same time it’s possible to write code that is compatible with both versions, and this is not by chance but a clear commitment of the <a href="https://www.toptal.com/python" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">developers</a> that drafted the several PEP (Python Extension Proposal). In the few cases in which syntax is incompatible, thanks to the fact that Python is a language with which we can dynamically modify code at runtime, <a href="https://bitbucket.org/gutworth/six/src/c17477e81e482d34bf3cda043b2eca643084e5fd/six.py?at=default#six.py-644" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">we can solve the problem </a>without relying on preprocessor with a syntax completely alien to the rest of the language.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The syntax is thus not a problem (especially ignoring versions of Python 3 before 3.3). The other big difference is the behavior of code, its semantics and the presence/absence of big libraries only for one of the two versions. This is indeed a significant problem, but it’s not completely unique or new for those who already have experience with other programming languages. You probably already happened to get an old codebase/library that fails to build with recent versions of the same compiler used originally. It’s the compiler itself in these cases that will help you (in Python, instead help will come from your own test suite).</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Why make the new version different then? What advantages will these changes bring to us?</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s assume we want to write a program to read the owner of files/directories (on a Unix system) in our current directory and print them on screen.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Does everything work correctly? Apparently it does. We specified the encoding for the file containing the source code, if we have a file created by олга (uid 1001) in our directory its name will be printed correctly, and even if we have files with non-ASCII names these will be printed correctly.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
There’s still a case that we haven’t covered yet though: a file created by олга AND with non-ASCII characters in the name…</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Let’s try to launch again our small script, and we’ll obtain a:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92217/toptal-blog-image-1457618686440-1bfc7deec12f42c68f5b74f6643270f7.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you think about it, a similar situation could be nasty: You have written your program (thousands of lines long instead of the few 4 of this example), you start to gather some users, some of them even from non-English speaking countries with more exotic names. Everything is okay, until one of these users decides to create a file that users with more prosaic name can create without any problem. Now your code will throw an error, the server might answer every request from this user with a error 500, and you’ll need to dig in the codebase to understand why suddenly these errors are appearing.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
How does Python 3 help us with this? If you try to execute the same script, you’ll discover that Python is able to detect right away when you’re about to execute a dangerous operation. Even without files with peculiar names and/or created by peculiar users, you’ll receive right away an exception like:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Related to line:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
The error message is even more easy to understand, in my opinion. The str object is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">owner</code>, and <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">node</code> is a bytes object. Knowing this, it’s obvious that the problem is due to the fact that <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">listdir</code> is returning us a list of bytes objects.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A detail that not everybody knows is that <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">listdir</code> returns a list of bytes objects or unicode strings depending on the type of the object that was used as input. I avoided using <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">listdir('.')</code> exactly to obtain the same behavior on Python 2 and Python 3, otherwise on Python 3 this would’ve been an unicode string that would’ve made the bug disappear.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If we try to change a single character, from <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">listdir(b'.')</code> to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">listdir(u'.')</code> we’ll be able to see how the code now works on both Python 3 and Python 2. For completeness, we should also change <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">'dario'</code> to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">u'dario'</code>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This difference in the behavior between Python 2 and Python 3 is however supported by a radical difference in how the two versions handle string types, a difference that is mainly perceived when porting from one version to the other.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In my opinion, this situation is emblematic of the maxim: “splitters can be lumped more easily than lumpers can be split”. What was lumped together in Python 2 (unicode strings and default strings of byte, which could be freely coerced together) has been split in Python 3.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
For this reason tools like <a href="http://python3porting.com/2to3.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">2to3</a>, even if well written and extremely useful to automate the conversion of every other difference, have some limitations. With the bytes/unicode split the difference in behavior surfaces at runtime, and a tool that can only do parsing/static analysis thus won’t be able to save you if you have a huge Python 2 codebase that mixes these two types. You’ll have to roll up your sleeves and properly design your API to decide if functions that until now accepted indiscriminately any type of strings should now work only with some of these (and which ones). Conversely, though getting a lot less use, tools of conversion from Python 3 to Python 2 have much easier life. Let’s see an example:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Sometime ago, I wrote a <a href="https://gist.github.com/berdario/39313b94cd08ebe6b3fb" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">toy HTTP server</a> (only dependency: <a href="https://pypi.python.org/pypi/python-magic/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">python-magic</a>), and this is the version for Python 2 (automatically converted from the Python 3 one without any need for manual changes): <a href="https://gist.github.com/berdario/8abfd9020894e72b310a" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">https://gist.github.com/berdario/8abfd9020894e72b310a</a></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Now, if you want you can have a look directly at the <a href="https://gist.github.com/berdario/34370a8bc39895cae139/1bb02892b3080c61a8e15c4d5b3be67a411e9318" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">code converted to Python 3</a> with 2to3, or you can convert it directly on your system. When trying to execute it you’ll realize how every error that you can try to fix by hand is related to the bytes/unicode split.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
You can manually apply changes like these: https://gist.github.com/berdario/34370a8bc39895cae139/revisions</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
And thus, you get your program working again on Python 3. These are not complex changes, but they require nonetheless to reason on which data types your functions are working upon, and on the control flow. It’s 13 lines of changes out of 120, a ratio not too easy to handle: with thousands of lines of code to port, you could easily end up with hundreds to modify.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92218/toptal-blog-image-1457618753801-12c3a16e83a63b082a2fc5c71c9f99aa.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
If you’re curious, you could then try to convert this code that you just brought to Python 3 back to Python 2. Using <a href="https://pypi.python.org/pypi/3to2" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">3to2</a> you’d obtain this: <a href="https://gist.github.com/berdario/cbccaf7f36d61840e0ed" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">https://gist.github.com/berdario/cbccaf7f36d61840e0ed</a>. In which the only change that had to be applied manually is <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">.encode('utf-8')</code> at line 55.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Starting from Python 3 (if you’ll ever need to convert it back to Python 2), it’s much easier. But if you need to have your code working on another version, a complete conversion like this is not the best choice. It’s much better to maintain compatibility with both versions of <a href="https://www.toptal.com/blog/tags/python" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">Python</a>. To do that you can rely on tools like <a href="http://python-future.org/futurize.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">futurize</a>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Even if you don’t have the chance to use Python 3 in production (maybe one of the libraries that you’re using is bulky and compatible with Python 2 only), I’d suggest for you to keep your code compatible with Python 3. You could even <a href="https://docs.python.org/3/library/unittest.mock.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">stub/mock</a> out the incompatible libraries, just so that you could run continuously <a href="https://tox.readthedocs.org/en/latest/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">your tests on both versions</a>. This will make it easier for you when in the future you’ll finally be ready to migrate to Python 3, not to mention how it can help you in better design your API, or to identify errors like in the example at the beginning of this post.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<img alt="" src="https://assets.toptal.io/uploads/blog/image/92219/toptal-blog-image-1457618777806-eaea65af8480534745fcb74b0dff1be4.jpg" style="border: 0px; box-sizing: border-box; display: block; margin: 0px auto 7px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" /></div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
All this talking about porting and byte/unicode difference, even if you were initially skeptical about using/starting with Python 3, probably led you to think of it as the lesser evil rather than tackling the porting in the future. But if porting is the stick, where’s the carrot? Is it the new features added to the language and to its standard library?</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Well, after 5 years of time from the release of the last minor version of Python 2, there are plenty of interesting tidbits that are piling up. For example I found myself relying quite often on things like the new <a href="https://www.python.org/dev/peps/pep-3102/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">keyword-only arguments</a>.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
When I wanted to write a function to merge an arbitrary number of dictionaries together (similar to what <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">dict.update</code> does, but without modifying the inputs) I found it natural to add a function argument to let the caller customize the logic. This way this function could be invoked as follows to simply merge multiple dictionaries by retaining values in the rightmost dicts.</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Likewise, to merge by adding the values:</div>
<div style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Implementing such an API in Python 2 would have required to define a <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">**kwargs</code> input and look for the <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">withf</code> argument. If the caller did mistype the argument as (e.g.) <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">withfun</code> the error would be silently ignored, though. In Python 3 instead it’s perfectly fine to add an optional argument after variable arguments (and it will be usable only with its keyword):</div>
<div class="embeddable_form-wrapper" data-role="blog_subscribe" data-view="blog_subscribe#form" style="border: 0px; box-sizing: border-box; color: #303030; font-family: 'Proxima Nova', Arial, sans-serif; font-size: 15px; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
<form action="https://www.toptal.com/blog/subscription" class="embeddable_form for-post" data-entity="blog_subscription" data-remote="" data-view="form#form" method="post" style="border-bottom-left-radius: 6px; border-bottom-right-radius: 6px; border-top-left-radius: 6px; border-top-right-radius: 6px; border: 0px; box-sizing: border-box; margin: 0px; min-height: 0px; min-width: 0px; padding: 30px 0px; vertical-align: baseline;">
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Since Python 3.5, the naive merging can actually be done with the <a href="https://www.python.org/dev/peps/pep-0448/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">new unpacking operator</a>. But even before 3.5 Python got an improved form of unpacking:</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This has been available to us since 3.0. Akin to destructuring, this kind of unpacking is a limited/ad-hoc form of the pattern matching commonly used in functional languages (where it is also used for flow control) and it’s a common feature in dynamic languages like Ruby and Javascript (where support for EcmaScript 2015 is available).</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
In Python 2, a lot of APIs that dealt with iterables were duplicated, and the default ones had strict semantics. Now instead everything will generate values as needed: <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">zip()</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">dict.items()</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">map()</code>, <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">range()</code>. Do you want to write your own version of <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">enumerate</code>? In Python 3 it’s as simple as composing functions from the standard library together:</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Is equivalent to <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">enumerate('abc', 1)</code>.</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Wouldn’t you like to define HTTP APIs as simply as this?</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
No more <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">'<int:user_id>'</code> ad-hoc syntax, and the ability to use any type/constructor (like <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">Decimal</code>) inside your routes without having to define your own converter.</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Something like this <a href="https://github.com/timothycrosley/hug" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">has already been implemented</a>, and what you see is valid Python syntax, exploiting the new annotations to make it more convenient to write APIs that are also self documenting.</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
These are just a couple of simple examples, but the improvements are far reaching and ultimately help you in writing more robust code. An example is exception chain tracebacks enabled by default, showcased in the aptly named post <a href="https://blog.ionelmc.ro/2014/08/03/the-most-underrated-feature-in-python-3/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">“The most underrated feature in Python 3”</a> by Ionel Cristian Mărieș, which is also covered in this <a href="http://migrateup.com/main-difference-python-3/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">other post</a> by Aaron Maxwell, together with the stricter comparison semantics of Python 3, and the new <code style="background-color: #fffffc; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(238, 238, 238); box-sizing: border-box; display: inline-block; font-size: 0.8em; line-height: 1em; margin: 0px; max-width: 100%; min-height: 0px; min-width: 0px; padding: 2px 5px; vertical-align: text-bottom;">super</code> behavior.</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This is not all. There are plenty of other improvements, these are the ones I feel have the most impact day-to-day:</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
A more thorough panorama can be obtained with the <a href="https://docs.python.org/dev/whatsnew/index.html" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">“What’s New”</a> pages of the documention, or for another overview of the changes I also suggest this other <a href="http://migrateup.com/whats-really-new-in-python-3/" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">post</a> by Aaron Maxwell and these <a href="https://speakerdeck.com/pyconslides/python-3-dot-3-trust-me-its-better-than-python-2-dot-7-by-dr-brett-cannon" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">slides</a> from Brett Cannon.</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
Python 2.7 will be supported <a href="http://legacy.python.org/dev/peps/pep-0373/#maintenance-releases" style="border: 0px; box-sizing: border-box; color: #3976cb; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;" target="_blank">until 2020</a>, but don’t wait until 2020 to move to a new (and better) version!</div>
<div style="border: 0px; box-sizing: border-box; font-size: 1.2em; line-height: 1.5em; margin-bottom: 1em; min-height: 0px; min-width: 0px; padding: 0px; vertical-align: baseline;">
This post originally appeared in <a href="https://www.toptal.com/python/python-3-is-it-worth-the-switch" target="_blank">Toptal Engineering blog</a></div>
<div class="embeddable_form-step is-email_form is-current" style="border: 0px; box-sizing: border-box; display: flex; flex-wrap: wrap; height: auto; margin: 0px; min-height: 0px; min-width: 0px; overflow: visible; padding: 0px; vertical-align: baseline;">
</div>
</form>
</div>
Anonymousnoreply@blogger.com1