#DataScience: 3 lessons from my first year working with data in NY

In this piece I share with you 3 lessons that I learned working at the company that uses data to solve the most complex problems of some of the world’s most important organizations.

This article is also available in Portuguese: #DataScience: 3 lições do meu 1º ano trabalhando com dados em Nova York

I recently completed a year working with data in New York for a Silicon Valley startup, at one of the most competitive markets of the world. I do not work creating complex algorithms – although I deal with this topic frequently – my role is in business development and deployment strategy. Making it clear that this is my perspective, I want to bring you some business lessons from large (huge!) data projects in the world’s most important organizations.

Lesson 1: General data integration usually causes more impact than localized use of super-algorithms and machine learning

The most common data problem in an organization is not understanding data, but integrating multiple sources and platforms. Even in a small startup, as were mine in Brazil, the information is spread across countless systems, environments and spreadsheets, such as Google Analytics, Mailchimp, Salesforce, Oracle, DB2, Google Spreadsheets, Microsoft Excel and many others. For a large (and for a small) organization, integrating all of this data into a single, centralized, environment – with proper permissions control – is more valuable than applying a machine learning algorithm to optimize only sales history, for example. There is a lot of inefficiency in the flow of information within a large organization, and inefficiency can be translated as money being burned. Improving the general flow of information within a corporation creates more value than optimizing only a small sector.

Lesson 2: <cliche>Data Science is about people</cliche>

In a large organization, you deal primarily with people, not machines and systems. It is people who make decisions based on what they see in the data, by applying a new super-sophisticated algorithm to a table with 1 billion records, the output must be simple enough to be understood and have enough value to change of habit of the person who will consume that information. In this sense, my experience with user interface and user experience has proven to be highly valuable, presenting well-constructed graphs, easy-to-understand control panels, and efficient applications, directly impacts how a person understands, evaluates, and recurrently uses a data science project.

Lesson 3: Changing habits requires huge energy investment

I learned much more about the success of data science projects by making software for the masses and reading (introductory material) on psychology and cognitive sciences, than by studying data science itself. Implementing a new data ecosystem in an organization – for example by integrating 28 data sources and 1 trillion records containing 10 years of historical information – is a huge job, but I would say that this is as big an effort as it is to make a small group – even if it is just a dozen people – with different professional backgrounds, quit their old habits, spreadsheets, emails, and move to a new workflow in a tool they have never seen. There is always the “but it was working”, and the challenge is to show that the new way will bring more productivity and precision to the team, freeing time from the staff to deal with other activities that an automated system still can not do.

The market is less sci-fi and more business

All of the human challenges involved in a used car sale are also involved in deploying a data science project. People have habits, preferences, power play, sympathy or antipathy and everything else that defines human relationships since we decided to get down from the trees and organize ourselves into groups. Understanding this fundamental aspect of business development can help you to be much more successful in your projects, be they data science or popsicle sales.

About the author:

• “Marco Gomes is a tech visionary”, Sarah Lacy on TechCrunch
• The Best Marketing Technologies Professional of the World, World Technology Network.
• 30 Under 30, Forbes.
• Series A investment from Intel Capital.
• ECOSOC Speaker, United Nations.