decor
 

Planet Big Data logo

Planet Big Data is an aggregator of blogs about big data, Hadoop, and related topics. We include posts by bloggers worldwide. Email us to have your blog included.

 

July 19, 2018


Revolution Analytics

Highlights from the useR! 2018 conference in Brisbane

The fourteenth annual worldwide R user conference, useR!2018, was held last week in Brisbane, Australia and it was an outstanding success. The conference attracted around 600 users from around the...

...
 

July 18, 2018

Big Data University

Scaling Our Private Portals with Open edX and Docker

Ever since we launched, Cognitive Class has hit many milestones. From name changes (raise your hand if you remember DB2 University) to our 1,000,000th learner, we’ve been through a lot.

But in this post, I will focus on the milestones and evolution of the technical side of things, specifically how we went from a static infrastructure to a dynamic and scalable deployment of dozens of Open edX instances using Docker.

Open edX 101

Open edX is the open source code behind edx.org. It is composed of several repositories, edx-platform being the main one. The official method of deploying an Open edX instance is by using the configuration repo which uses Ansible playbooks to automate the installation. This method requires access to a server where you run the Ansible playbook. Once everything is done you will have a brand new Open edX deployment at your disposal.

This is how we run cognitiveclass.ai, our public website, since we migrated from a Moodle deployment to Open edX in 2015. It has served us well, as we are able to serve hundreds of concurrent learners over 70 courses every day.

But this strategy didn’t come without its challenges:

  • Open edX mainly targets Amazon’s AWS services and we run our infrastructure on IBM Cloud.
  • Deploying a new instance requires creating a new virtual machine.
  • Open edX reads configurations from JSON files stored in the server, and each instance must keep these files synchronized.

While we were able to overcome these in a large single deployment, they would be much harder to manage for our new offering, the Cognitive Class Private Portals.

Cognitive Class for business

When presenting to other companies, we often hear the same question: “how can I make this content available to my employees?“. That was the main motivation behind our Private Portals offer.

A Private Portal represents a dedicated deployment created specifically for a client. From a technical perspective, this new offering would require us to spin up new deployments quickly and on-demand. Going back to the points highlighted earlier, numbers two and three are especially challenging as the number of deployments grows.

Creating and configuring a new VM for each deployment is a slow and costly process. And if a particular Portal outgrows its resources, we would have to find a way to scale it and manage its configuration across multiple VMs.

Enter Docker

At the same time, we were experiencing a similar demand in our Virtual Labs infrastructure, where the use of hundreds of VMs was becoming unbearable. The team started to investigate and implement a solution based on Docker.

The main benefits of Docker for us were twofold:

  • Increase server usage density;
  • Isolate services processes and files from each other.

These benefits are deeply related: since each container manages its own runtime and files we are able to easily run different pieces of software on the same server without them interfering with each other. We do so with a much lower overhead compared to VMs since Docker provides a lightweight isolation between them.

By increasing usage density, we are able to run thousands of containers in a handful of larger servers that could pre-provisioned ahead of time instead of having to manage thousands of smaller instances.

For our Private Portals offering this means that a new deployment can be ready to be used in minutes. The underlying infrastructure is already in place so we just need to start some containers, which is a much faster process.

Herding containers with Rancher

Docker in and of itself is a fantastic technology but for a highly scalable distributed production environment, you need something on top of it to manage your containers’ lifecycle. Here at Cognitive Class, we decided to use Rancher for this, since it allows us to abstract our infrastructure and focus on the application itself.

In a nutshell, Rancher organizes containers into services and services are grouped into stacks. Stacks are deployed to environments, and environments have hosts, which are the underlying servers where containers are eventually started. Rancher takes care of creating a private network across all the hosts so they can communicate securely with each other.

Schematic of how Rancher is organized

Getting everything together

Our Portals are organized in a micro-services architecture and grouped together in Rancher as a stack. Open edX is the main component and itself broken into smaller services. On top of Open edX we have several other components that provide additional functionalities to our offering. Overall this is how things look like in Rancher:

A Private Portal stack in Rancher

There is a lot going on here, so let’s break it down and quickly explain each piece:

  • Open edX
    • lms: this is where students access courses content
    • cms: used for authoring courses
    • forum: handles course discussions
    • nginx: serves static assets
    • rabbitmq: message queue system
  • Add-ons
    • glados: admin users interface to control and customize the Portal
    • companion-cube: API to expose extra functionalities of Open edX
    • compete: service to run data hackathons
    • learner-support: built-in learner ticket support system
    • lp-certs: issue certificates for students that complete multiple courses
  • Support services
    • cms-workers and lms-workers: execute background tasks for `lms` and `cms`
    • glados-worker: execute background tasks for `glados`
    • letsencrypt: automatically manages SSL certificates using Let’s Encrypt
    • load-balancer: routes traffic to services based on request hostname
    • mailer: proxy SMTP requests to an external server or sends emails itself otherwise
    • ops: group of containers used to run specific tasks
    • rancher-cron: starts containers following a cron-like schedule
  • Data storage
    • elasticsearch
    • memcached
    • mongo
    • mysql
    • redis

The ops service behaves differently from the other ones, so let’s dig a bit deeper into it:

Details of the ops service

Here we can see that there are several containers inside ops and that they are usually not running. Some containers, like edxapp-migrations, run when the Portal is deployed but are not expected to be started again unless in special circumstances (such as if the database schema changes). Other containers, like backup, are started by rancher-cron periodically and stop once they are done.

In both cases, we can trigger a manual start by clicking the play button. This provides us the ability to easily run important operational tasks on-demand without having to worry about SSH into specific servers and figuring out which script to run.

Handling files

One key aspect of Docker is that the file system is isolated per container. This means that, without proper care, you might lose important files if a container dies. The way to handle this situation is to use Docker volumes to mount local file system paths into the containers.

Moreover, when you have multiple hosts, it is best to have a shared data layer to avoid creating implicit scheduling dependencies between containers and servers. In other words, you want your containers to have access to the same files no matter which host they are running on.

In our infrastructure we use an IBM Cloud NFS drive that is mounted in the same path in all hosts. The NFS is responsible for storing any persistent data generated by the Portal, from database files to compiled static assets, such as images, CSS and JavaScript files.

Each Portal has its own directory in the NFS drive and the containers mount the directory of that specific Portal. So it’s impossible for one Portal to access the files of another one.

One of the most important file is the ansible_overrides.yml. As we mentioned at the beginning of this post, Open edX is configured using JSON files that are read when the process starts. The Ansible playbook generates these JSON files when executed.

To propagate changes made by Portal admins on glados to the lms and cms of Open edX we mount ansible_overrides.yml into the containers. When something changes, glados can write the new values into this file and lms and cms can read them.

We then restart the lms and cms containers which are set to run the Ansible playbook and re-generate the JSON files on start up. ansible_overrides.yml is passed as a variables file to Ansible so that any values declared in there will override the Open edX defaults.

Overview of file structure for a Portal

By having this shared data layer, we don’t have to worry about containers being rescheduled to another host since we are sure Docker will be able to find the proper path and mount the required volumes into the containers.

Conclusion

By building on top of the lessons we learned as our platform evolved and by using the latest technologies available, we were able to build a fast, reliable and scalable solution to provide our students and clients a great learning experience.

We covered a lot on this post and I hope you were able to learn something new today. If you are interested in learning more about our Private Portals offering fill out our application form and we will contact you.

Happy learning.

The post Scaling Our Private Portals with Open edX and Docker appeared first on Cognitive Class.

Making Data Meaningful

Hey You … Get Out of My Cloud!

cloudDo you remember these recent stories?  On July 31, 2012 Dropbox admitted it had been hacked. (Information Week, 8/1/2012).  Hackers had gained access to an employee’s account and from there were able to access LIVE usernames and passwords which could allow them to gain access to huge amounts of personal and corporate data.  Just four days later, Wired® writer Mat Honan’s Twitter account was hacked via his Apple and Amazon accounts (story in Wired and also reported by CBS, CNN, NPR and others).

Did you notice the common theme behind these reports?  Hackers didn’t get through the defenses of the Cloud by brute force.  Instead, they searched out weak points and exploited other vulnerabilities led to by those entry points.  In these examples – as in countless others – the weak points were processes and people.

The Dropbox hack was made possible by an employee using the same password to access multiple corporate resources, one of which happened to be a project site which contained a “test” file of real unencrypted usernames and passwords.  Either one could be considered a lapse in judgment – I mean, who thinks it is a good idea to store unencrypted user access information on a project site??? – but added together, these lapses made a result much more dangerous than the sum of their parts.

Mat Honan’s hack was made possible in part by process flaws at large and popular companies.  Again, each chink taken individually would likely not have been as damaging as the series of flaws building on each other.  Apple or Amazon individually didn’t provide enough information for hackers to take over Mr. Honan’s account, but taken together their processes and individual snippets of data provided the opportunity.

My purpose in writing this isn’t to scare anyone away from the Cloud or its legitimate providers.  The Cloud is cost-effective, portable, scalable, stable, and here to stay.  And it is as secure as technology will allow.  But as these stories illustrate, technology isn’t the risk.  Information wasn’t compromised by brute-force hacking or breaking encryption algorithms.  Data was put at risk by people and processes.

Have you ever worked with someone who messed up something royally by not following a documented process?  Or do you know someone who clicked a link in a bogus email and infected their laptop – or even the whole company – with a virus?  They might be working for your Cloud provider now.  Don’t rely on those folks to protect your data in the Cloud.  Instead, protect it yourself with Backups, Password Safety and Data Encryption before entrusting your precious data to the Cloud.  If a hacker gets into your Cloud, at least you won’t be the easiest target.

The post Hey You … Get Out of My Cloud! appeared first on Making Data Meaningful.

Making Data Meaningful

Real-Time Analytics

Real Time AnalyticsIf you have ever shopped at Amazon you may have noticed a “Featured Recommendations” section that appears after your initial visit. These recommendations get automatically updated after the system notices a change in the shopping pattern of a particular member. This is real-time analytics at work. The system is using the data at hand and coming up with suggestions in near real-time. With more companies investing into a mobile business intelligence initiative, real-time analytics is an essential requirement to ensure a good return on investment.

I think that the implementation of a solution to get real-time analytics could be a costly endeavor. This would require implementation of technologies like Master Data Management and delivery options like cloud and/or mobile BI. Cloud BI presents its own set of security concerns, which is why some of the region’s largest companies are hesitant to implement such a solution. According to one BI manager, the company’s executives do not support the notion of putting their data into the cloud without the implementation of certain security measures. Their need for a mobile BI strategy would require security that would enable the company to delete everything from a device if it is stolen or misplaced.

Insurance companies and retail stores can greatly benefit from such technology. The off-site sales reps will be able to see current information about potential customers including updated life changing events right on their mobile devices, which would increase the likelihood of either gaining a new customer or retaining an existing one*. In-store managers at grocery stores can get a real-time report about slow moving items allowing them to increase sales by changing displays. Real-time analytics can be on-demand where the system responds to a certain request by an insurance sales rep or it can be a continuous hourly report to the store manager of a grocery store**.

Overall, real-time analytics gives a company a competitive advantage over its rivals but requires heavy investment into the implementation of the technology and the guarantee of proper security measures being put in place with delivery options like the cloud. This information is helpful for quick decisions, but companies should still make all major decisions by looking at historical data and studying the trends.

Sources:

*Pat Saporito, “Bring your Best”, Best’s Review, September 2011

**Jen Cohen Crompton, “Real-Time Data Analytics: On Demand and Continuous”.  

The post Real-Time Analytics appeared first on Making Data Meaningful.

Making Data Meaningful

Business Intelligence Adoption: Goal Setting

BI AdoptionTypically, strategic goals start off as high-level initiatives that involve revenue-based targets.  Revenue targets are followed up with operational efficiency goals (or ratios) that keep expenses in line and improve profit margins.  These goals and ratios serve as the ultimate yardstick in measuring top-end strategic performance.  There may also be competitive goals that utilize different measures such as market share, product perception, etc.   Companies believe they can achieve these results based on internal and external competitive factors.  It is important to note that the internal and external factors typically drive the timing and define the tactical activities that will be employed to achieve results.

For example, a change in government regulation may present a significant opportunity for the company that is first to capitalize on the change.  An example of an internal factor may be outstanding customer service that can serve as a market differentiator to attract and retain customers.

These competitive factors and performance measures drive the definition of the tactical operations (or plan) needed to achieve strategic goals. Tactical operations are ultimately boiled down to human activities and assigned to managers and their employees.  Human activities impact revenue, profit, and quality.  Even quality activities ultimately impact revenue and profit.

Example, an insurance company may excel at gathering high quality claims data that results in lower claim expenses and legal costs.

Human activities are incorporated into an individual’s performance plan.  Before defining the human activities though, the goals, competitive factors, and tactical operations need to be gathered into a data repository.  Once gathered, they will be used to gain and communicate corporate alignment.

Depending on your role in the organization, you may be called upon to help define and capture the financial performance ratios.  You may also be responsible for gathering and storing external factors such as survey results, industry statistics, etc.

If all goes well, the corporation captures the revenue and performance goals and defines how performance is to be measured.  This is also communicated across the enterprise (gaining alignment).  The performance goals and target financial ratios can be stored in the corporation’s data repository.  The measuring and communicating of progress will be accomplished using a company’s reporting toolset.  The company has to decide the best frequency to communicate actual performance compared to stated goals.  This frequency can be daily, weekly, monthly, or quarterly with the emphasis on providing continual feedback.  Reporting on performance results is the first, and most basic, step in the adoption of BI practices.  Performance reporting answers the question “What happened?” (Davenport & Harris, 2007).  It is very important but only the first step.

  • Davenport, T. H., & Harris, J. G. (2007). Competing on Analytics The New Science of Winning. In T. H. Davenport, & J. G.
  • Harris, Competing on Analytics The New Science of Winning (p. 8). Boston: Harvard Business School Press.

The post Business Intelligence Adoption: Goal Setting appeared first on Making Data Meaningful.


Datameer

Five Best Practices for Software Maintenance

In this blog, we cover five best practices for system administrators to keep users satisfied when it comes to maintenance updates: schedule, think holistically, review urgency, test changes...

...
 

July 17, 2018

Making Data Meaningful

Forecasting and Predictive Analytics

Wikipedia defines Forecasting as the process of making statements about events whose actual outcomes (typically) have not yet been observed.

Examples of forecasting would be predicting weather events, forecasting sales for a particular time period or predicting the outcome of a sporting event before it is played.

Wikipedia defines Predictive Analytics as an area of statistical analysis that deals with extracting information from data and using it to predict future trends and behavior patterns.

Examples of predictive analytics would be determining customer behavior, identifying patients that are at risk for certain medical conditions or identifying fraudulent behavior.

Based on these definitions, forecasting and predictive analytics seem to be very similar…but are they? Let’s break it down.

Both forecasting and predictive analytics are concerned with predicting something in the future, something that has not yet happened. However, forecasting is more concerned with predicting future events whereas predictive analytics is concerned with predicting future trends and/or behaviors.

So, from a business perspective, forecasting would be used to determine how much of a material to buy and keep on stock based on projected sales numbers. Predictive analytics would be used to determine customer behavior like what and when are they likely to buy, how much do they spend when they do buy, and when they buy one product what else do they buy (also known as basket analysis).

Predictive analytics can be used to drive sales promotions targeting certain customers based on the information we know about their buying behavior. Likewise, the information obtained from predictive analytics can be used to influence sales projections and forecasting models.

Both, predictive analytics and forecasting, use data to achieve their purposes. But, it’s how they use that data that is much different.

In forecasting, data is used to look at past performance to determine future outcomes. For instance, how much did we sell last month or how much did we sell last year at this time of year. In predictive analytics, we are looking for new trends, things that are occurring now and in the future that will affect our future business. It is more forward looking and proactive.

So, although forecasting and predictive analytics are similar and closely related to one another, they are two distinctively different concepts. In order to be successful at either one, you have to have the right resources and tools in place to be able to extract, transform and present the data in a timely manner and in a meaningful way.

A common problem in business today is people spend much more time preparing and presenting information than they do actually determining what the data is telling them about their business. This is because they don’t have the right resources and tools in place.

At Making Data Meaningful we have the resources, strategies and tools to help businesses access, manage, transform and present their data in a meaningful way. If you would like to learn more about how we can help your business, visit our website or contact us today.

The post Forecasting and Predictive Analytics appeared first on Making Data Meaningful.

Making Data Meaningful

MicroStrategy: Scalable Yet Agile

Are you looking for an analytics tool that is simple enough to get up and running fast and has the capability to keep up with your company as its Business Intelligence requirements mature? If so, you will want to check out the new offerings by MicroStrategy.

Industry Leading Analytics – Enterprise Capable

MicroStrategy has long been known for its large scale enterprise reporting and analytics solutions.  They have also been the leaders in offering analytics on the mobile platform.

MicroStrategy’s best-in-class capabilities have traditionally been expensive to purchase and require expert technical assistance to implement and maintain. Large organizations are able to realize economies of scale but small and medium sized companies may find it difficult to justify an initial large investment in software, resources, and infrastructure.

To overcome an initial software investment, MicroStrategy does offer a free enterprise version of its analytics suite for up to 10 users, called Analytics Suite.  This 10-user license provides the opportunity to try-before-you-buy before rolling out to the larger enterprise.  This product can be hosted on-premise or on MicroStrategy’s cloud.

Companies still have to develop internal resources to handle security, data architecture, metadata, and reporting requirements.

Competition from the Edges

In recent years companies like MicroStrategy, Cognos, SAP, and Oracle have lost ground to smaller, more agile, startups like Tableau and Qlikview. The newer companies have made it faster (and easier) to get up and running with appealing visualizations.

These smaller companies are now trying to make their products scalable with respect to handling complex security, data architecture, and metadata requirements that are part of all mid- to large-sized implementations.

MicroStrategy’s Response

MicroStrategy has responded to the competition by offering two smaller-scale solutions that can be implemented in a matter of weeks: Analytics Desktop and Analytics Express.

Personal-Sized Analytics

As its name implies, Analytics Desktop is installed on an individual’s computer.  This product can attach to a variety of relational, columnar, and map reduce databases.  It can also attach to Excel worksheets and SalesForce data.  Analytics Desktop is designed for individual data discovery, possesses some advanced analytics capabilities, and can publish personal interactive dashboards.  Data sharing is limited to distribution via PDF, Spreadsheet export, image files, and distributable flash files.  Best of all, Analytics Desktop is free and offers free online training.

Department-Level Solution

Analytics Express has all of the features of Desktop except that it is hosted completely in MicroStrategy’s cloud environment.  There are no internal MicroStrategy hardware requirements.  A secure VPN connection between MicroStrategy’s cloud and your company’s firewall can be configured to protect your data.  The cloud-based analytics solution can import data from your organization’s back-end databases and refresh the data on a regularly scheduled basis.  Importing the data provides the benefit of much improved analytics performance and data availability.

It’s a Mobile World

Additional visualization options are available plus the ability to deploy solutions tailored for the iPad.  Access to Drop Box and Google Drive are also available.

Enterprise-Level Security

Security features include user authentication, single-sign-on, user and user group management, row-level security, dashboard level security, and user-based data filtering.

Analytics Everywhere

Dashboards can be embedded in other web pages or on intranet sites.  Visualizations can also be scheduled for email distribution.

Deploy Before You Buy

Organizational risk is minimized because MicroStrategy offers a free one-year trial of Analytics Express.  With all architecture hosted in the cloud, your organization won’t have to belly up any hardware or technical resources to support this product either.

Agile and Scalable

Both the Desktop and Express editions benefited from an improved web-based user interface designed to make the creation of dashboards easier.  MicroStrategy also leveraged its extensive portfolio of Enterprise-Level features by making them available in the hosted solution.  This ensures that MicroStrategy can meet the ever-evolving Business Intelligence and Analytics needs of your organization.

The post MicroStrategy: Scalable Yet Agile appeared first on Making Data Meaningful.

Making Data Meaningful

Organizing Large Projects – How to Avoid “Death by Meeting”

When I first heard the expression “Death by Meeting”, I thought it was the latest Stephen King novel, but after being the project manager of a project where I was expected to be involved in 20 meetings per week, dying seemed like a welcome alternative.  You can avoid this slow, painful death by creating a project structure that focuses efforts and communications and reduces meetings.

In addition to the typical project management issues associated with the multitude of tasks required for large projects, there is a significant challenge in creating an efficient, effective project structure that drives the project effort to the correct worker-bee level and enables good project status communications, but streamlines the number of meetings required to achieve these goals.  One approach that has worked for me is the use of Project Workgroups.

Most large projects consist of numerous tasks that can usually be grouped together in some manner.  These groupings may be by departmental function (Finance, IT, Purchasing, etc.) by activity (sales, development, implementation, training, etc.), by deliverable (software release, management reporting, etc.), or perhaps some other logical division.  Regardless of the grouping, there will be common goals and activities that will enable creation of workgroups reflecting these goals.

Once you have determined some logical workgroups, the next step is to define a project team structure.  At the top of the structure is the Steering Committee.  This is the group that is made up of senior management who are the key stakeholders for the project.  The role of this group is to provide high-level direction, provide resources (monetary and personnel), and resolve major roadblocks to the success of the project.  Steering Committees may oversee multiple concurrent projects, and will meet on a monthly or quarterly basis.

At this level, the Steering Committee members want to know where the project stands in terms of schedule, budget, and final deliverables.  A fantastic tool for providing the Steering Committee this information is via a project dashboard.  This dashboard should consist of a few key measurements with a status of each, using easy-to-read indicators like traffic lights or gauges.  Here is an example:

This dashboard eliminates the need for developing voluminous detailed reports, and provides for exception level discussions.  Only items that are yellow or red require explanation, so meetings are focused and their lengths are minimized.

The next level down from the Steering Committee is the Project Management Team, sometimes referred to as the Project Core Team.  This team consists of key middle-management personnel representative of the primary functional areas affected by the project.  The Core team should meet weekly or bi-weekly and is responsible for the direct management of the project activities.  The RAID (Risks, Action Items, Issues, Decisions) document I referenced in my previous blog is the perfect communications tool for the Core Team.  It provides a clear, concise mechanism for letting the team members see the critical items that require their attention.

The next level of the project organization below the Project Core Team contains the working groups for the project.  The makeup of the workgroups will vary by project; however, this is the level where the daily tasks of the project are managed.  This is the level that can bring you closest to a near-death experience since the number of teams and meetings is highest here.

Analyze your project and its deliverables to determine the best method for defining the workgroups.  An excellent place to start is with the desired deliverables since it is difficult to split a single deliverable across workgroups.  Another factor to consider is inter-departmental dependencies.  Departments that closely interact with each other and/or are dependent upon each other can be combined on a workgroup to leverage that interdependency.

Meetings at this level of the project team need to be at least weekly.  As above, the RAID document can be used to focus and track activities of the group, and facilitate communications to the project manager and the Project Core Team.  If the tracking and reporting mechanism is standardized, then the project manager does not have to participate in all of these meetings.  Focus the workgroups on the RAID documents and they will drive the agendas and reports so that meeting death takes a holiday!

In summary, to avoid the prospect of having the next project you manage being the planning of your own funeral after a painful “death by meeting” experience, try using the techniques described in this article.  By constructing a project team structure as described, you can keep all the affected parties updated, involved, and focused in a manner that streamlines communications, maximizes resources, and minimizes wasteful meetings.  The use of standardized task tracking and reporting tools will enable you as project manager to have visibility of all the project workgroups’ activities, and provide you the tools necessary to drive the project home successfully.

The post Organizing Large Projects – How to Avoid “Death by Meeting” appeared first on Making Data Meaningful.

Making Data Meaningful

Internet of People: An Analysis

I recently read an article by Strategy& on the Internet of Things (IoT) entitled “A Strategist’s Guide to the Internet of Things”.  This article begins with the current state of electronic world.  There will be 50 billion Internet responsive devices by 2020, and only a third will be smart devices. The rest will be…well, hard to put into a single word.  They will range from smart appliances to RFID (Radio-Frequency Identification) chips and populate nearly everything so long as there is meaningful information worth mining.  According to the article, there are three broad strategic categories within the IoT.  There are enablers, enhancers, and engagers.  Enablers generate and instate underlying technology; Engagers deliver IoT to customers through a streamlined one-stop shop; Enhancers concoct value-added services for engagers. These categories are trying to approach a very complicated problem with structure. Yet, structure implies a foundation—and in reading this article I came upon a cliché but logical question, “Is there an Internet of People (IoP)?”  My question, after a brief skim was, of course, answered.  Yes, the concept was used, but in a different manner than I had anticipated.  The victor to this vanguard saw the IoP as a shift in the way government and economic models operate under the IoT, and he approaches this question philosophically, The social and economic contracts between people, businesses and governments are undergoing a fundamental change and new rules of the game and governance models are needed for the future digital societies”.

This same thought that had piqued my interest had also led another to the same conclusion: the IoP is the foundation of the IoT.  Sensors and data collectors respond to products, and products respond to people.  An example can be seen in current political revolutions: In 2010, Hong Kongians displayed disfavor against anti-democratic sentiment from Mainland China through texting, twitter, and Facebook (The Economist: Protest in Hong Kong). When China made such communication impossible in recent protests by disconnecting communication infrastructure; protesters turned to FireChat, a texting app that operates in short range without Wi-Fi.  The irony of this situation is simple: humans used democracy to support democracy through technology.  Most recently, an article about Iran by the Economist, in the most prescient sense, assessed Iran through saying: “The revolution is over.” Yet, in many ways, the revolution in Iran has just begun.  As the article reports, “So-called VPNtrepreneurs sell software and access codes to bypass controls. E.G. a 21-year-old, who resells software, says he charges a dollar a month or $10 a year to his 80,000 clients and he uses his day job at an IT company as a cover. And, occasionally he pays the cyber-police a few hundred dollars in bribes.” No longer can Ruhollah Khomeini rule as absolute dictator.  Power has been decentralized.  This lovely article was an honorable Facebook post of mine: The point being, that people are people and each person is different.  This is the irony of the IoT-for once mass production has turned to personalization.

According to The Economist, Amazon.com is sensational because it sells over 230M products that are each accessible within seconds. Even the concept of Amazon appears decentralized.  It doesn’t matter where you are; it will be there quick and, in most cases, relatively cheap.  It is important to recognize that this decentralization has been underway for a long while.  The past two decades have proved that people can work from home without losing connection.  This is amazing.  I remember watching Austin Powers when I was younger and gapping at the videoconference scene between Dr. Evil and World Leaders.  I thought, “What if that was real?” In 2013, FaceTime mass-produced this reality as a Voice over IP (VoIP).  Soon Facebook, Gmail, and other messaging services also made this free.  Even though, Skype has been doing this for a decade.  I took advantage of these developments while I studied in London last year, and in many ways, I felt much closer to home that the “pond” used to suggest. So close, in fact, that I didn’t come home once during that period.  Connectivity, essentially, has changed the way that we communicate.  Price point no longer inhibits the amount of time we spend on long distance calls.  This increase in connectivity options has literally come from an increase in connectivity. The Economist recently posed an honest question on hotspots, “As Wi-Fi proliferates, who needs cellular wireless?” By 2018 the number of public hotspots is expected to increase from 47M to 340M.  The point being, that technology has changed human behavior.  It should be no surprise that communication companies make up the largest companies.  Technocracy, a term developed by Californian engineer, William Henry Smyth, in 1919, rules the world.  Jack Ma, Mukesh Ambani, Elon Musk, Steve Jobs, Bill Gates, Larry Ellison – these are the famous men of today, sorry Jamie Dimon but you’re a rare example, and they are famous because they are accentuating our most important human activity: communication.

This is why the IoP, through the foundation of techno-communication, is being backed into by the IoT.  As we push against products and they push against us—An Internet of People is going to include the customization of not only our products but of ourselves.

The post Internet of People: An Analysis appeared first on Making Data Meaningful.

Making Data Meaningful

Blobitty Blob Blob Blob….

At my current client, we are working on a major rewrite of their Claims Management System, which has included moving the old data from an Oracle database to a new SQL database being designed by our team. Part of this conversion involved a particular challenge. We were tasked with extracting all of the documents stored in the Oracle DB as BLOBs (HUGEBLOBs to be exact) and loading them out to the file system. The documents are going to be stored in the new SQL database in a “File Table”. A File Table is essentially an index to these files, while the files themselves are not physically stored in the database, they are stored out in the file system. The File Table contains certain fields including a unique stream_id, a path_locator that indicates where the document is, and other pertinent information about it, including the file_stream encoding, file name, file type, file size and other attributes. There is no ability to add additional fields to the built-in SQL File Table structure for storing other key information.

There are 1.2 million documents that have to be extracted, saved to the file system, and linked to the appropriate claimant’s record in the new database. Without being able to attach any other fields to these documents while they are extracted to the file system to link the document to the appropriate client record, that is challenge #1. Challenge # 2 is getting the documents out en masse (certainly we couldn’t save them out one by one using the existing front end application). Challenge #3 comes later, and I’ll get to that.

So I did some searching and found a built-in Oracle function called UTL_FILE that extracts the documents from the Oracle BLOB table. I tweaked the function into a script that contains a cursor to select the documents in chunks at a time, then loops for each document, renames the document, and runs UTL_FILE to save it out—looping each file into further chunks when they are too big for one write (the limit is 32k bytes at a time). This script then writes them out to the file system on the network. This solved Challenge #2 which was to bulk extract all of these documents, although due to the time cost of the script, they still will need to be done in smaller batches at a time.

Renaming the documents as they came out of the system solved Challenge #1, where these documents needed to somehow be linked to the original claimant’s record. I appended the original Claimant ID number to the front of the document name, separated by “—“ so it would be easy to use SUBSTRING to get the ID number out later for loading into the table that links the File Table records (by the stream_id) to the Claimant records.

Challenge #3 came when trying to open some of the documents after they were exported. They were corrupted—it was approximately 10% of the documents. After a lot of head scratching and looking for similar patterns in these documents to explain why they were corrupted yielded no clear answer. A suggestion from my manager led me to explore the code from the old application. I looked at the code that uploaded documents into to the Oracle table, and the code that opened them and allowed the users to export and save them out to the file system. Therein lied the answer. It appeared that some of the documents were being compressed before being uploaded to the Oracle database. There really seemed no business rule for which were to be compressed, and there was no indicator in the database as to if they were compressed or not. Therefore, I altered my UTL_FILE script to uncompress all of the files before saving them out. Unfortunately, if a file was NOT compressed, it would throw an error. So, I again altered my script to catch the error, and NOT uncompress those documents. Voila, the script worked like a charm. Here it is in all its glory, and our customer is happy that we can get all their BLOBs out!

The post Blobitty Blob Blob Blob…. appeared first on Making Data Meaningful.


Revolution Analytics

Video: R for AI, and the Not Hotdog workshop

Earlier this year at the QCon.ai conference, I gave a short presentation, "The Case for R, for AI developers". I also presented an interactive workshop, using R and the Microsoft Cognitive Services...

...
Ronald van Loon

Digital Meets 5G; Shaping the CxO Agenda

The age of technology is way past its nascent stage and has grown exponentially during the last decade. In my role, travelling to events and meeting with thought leaders I am aware of the developments made across different technological fronts. During these events I have had the opportunity to meet and greet some of the brightest marketing minds. One such person, who is playing an imperative role in ensuring the smooth run of things in both Ericsson and across the digital sphere, is Eva Hedfors.

Eva Hedfors, who is the Head of Marketing and Communications at Ericsson Digital Services, is a leading driver in evolving the perception of Ericsson as a partner to operators transformation from Communication Service Providers to Digital Service providers. I had the opportunity to first meet Eva during Ericsson studio tour in Kista, Sweden. During our first meeting I could tell of her knowledgeable insights and the positive vision she had. Just recently I had the opportunity of interviewing her for a topic she will be presenting in a webinar on the 20th of June. The topic of the webinar – Digital Meets 5G; Shaping the CXO Agenda – is up for interpretation, and she did give me details regarding what she is expecting from the webinar, and how she plans to go about answering some of the questions in this regard.

What Steps should be taken by CxO’s for a Smooth Transition to 5G?

Eva shared her insights on how CxOs could prepare for a smooth transition to Digital Service Providers powered by 5G.”The initial 5G technology deployment will target the massive growth of video traffic in networks, but a leading and hard-to-crack requirement for all CxO’s is also to realize the industry potential and find new business growth through 5G. This involves to both innovate and participate in eco-systems, as well as to optimize the route for marketing such 5G services. CxO’s  can take advantage of 5G to address relatively new segments and industry use cases in mission critical IoT as well as Massive IoT.” Eva explained the business models one creates also needs to be up-to-date and should reflect what’s happening in the market. Since the plan for 5G is rather new, most companies and industries won’t know much about it. Hence, it is necessary that decision makers in Telcos to position their existing capabilities towards different industries and using Network Slicing is one way to do that already on 4G.  To capture the potential in 5G, for many CxO’s means focus to create a revamped strategy for billing and charging systems into a Digital Business Support System (BSS). Moreover, a proper infrastructure needs to be provided to ensure that the end consumer gets to experience the technology in a seamless manner. This would help generate positive insights. 5G is here today, and action needs to start from right now!

How to Avoid the Challenges Involved in Digitization?

The first step to avoiding the challenges involved in digitization is to recognize the efforts most customers have to put in place when engaging with their Telco provider. Once these efforts have been quantified, Telcos can take the necessary action. For the customers, touch points should be made accessible, and there should be no hindrance in communication for B2C, B2B and B2B2C customers. Failure to put the right digital IT infrastructure in place, including analytics and Digital BSS, will limit the business potential of 5G. That is why 5G and Digitalization needs to be planned and executed not as individual technology transformation projects, but as one transformation that aligns towards the same overall business objective in each time frame.  Moreover, the technology teams should be motivated to simplify the core network and make it programmable. Eva mentioned that it was imperative for organizations to start already now and simplify the journey from vEPC to 5G Core for proper implementation and monetization of these revamped services.

Research for 5G Readiness across Different Industries

When asked about the research done to analyze the 5G readiness across different industries, Eva mentioned that Ericsson has done several reports on the potential of 5G across industries.

  1. The 5G business potential study by Ericsson analyzes the business opportunities that come from proper industrial digitalization. The report focuses on the opportunities for organizations present in 10 of the major industries including, Manufacturing, Energy and Utilities, Automotive, Public Safety, Media and Entertainment, Healthcare, Financial Services, Public Transport, Agriculture and Retail. There are detailed use cases for these industries present in the research, which may help stakeholders in these industries to make a decision regarding 5G usage.
  2. Another research based study released by Ericsson in this regard is the guide to capturing 5G-IoT Business Potential. The study answers questions pertaining to the selection of industries and what use cases to address. The insights have been collected from over 200 5G use cases that illustrate how operators can start their IoT business now through the use of 5G.

How Can 5G Technology Improve the Customer Experience Offered to existing Customers by Service Providers?

Enhanced Mobile Broadband is one of the major benefits of 5G technology, according to Eva, and it will help service providers enhance the experience they offer to their customers, who continue to increase consuming video on mobile devises . Better performance, reliability and ultra-high speed are some of the examples of the broadening consumer experience that can be provided through the 5G experience. According to a recent ConsumerLab report conducted by Ericsson, more than 70 percent of all consumers identify performance as one of the major expectations they are looking forward to from 5G.

.

What are the Preparations and Biggest Challenges for 5G Readiness?

Through our industry partnerships we do know, many organizations across many industries have started to analyze how 5G will help drive their digital transformation. 5G business models are being crafted to ensure that the implementation is as smooth as possible. The biggest challenge to capturing the market potential for all actors in the industrial eco-systems, including telecom operators, is the investment in technology and business development. Business development will fall along the lines of organizational adaptation, and Eva believes that a proper infrastructure needs to be provided. It is necessary that 5G be provided the right infrastructure for industry wide implementation. Only organizations that have created the right structure and the model required for 5G implementation are ready for the technology. Without organization-wide infrastructure, 5G would be just like a car running without roads and filling stations.

Integrating 5G Technology across Infrastructure

Like we have talked about above, decision makers need to realize the importance of a proper automated structure that spans across all touch points to ensure that there is no hindrance to 5G services adoption. To that end, organizations also need to realize the importance of an architecture evolution strategy. The evolution strategy should seamlessly integrate 5G across the infrastructure and ensure the full flexibility in the handling of billing, charging and customer interaction.

Both IoT and 5G technologies are shaping the digital transformation and transforming all digital architecture by helping organizations evolve their services and infrastructure. 5G particularly brings a new level of characteristics and performance to the mix, which will play an important role in the digitalization of numerous industries. Telecom operators leveraging the power of 5G technologies can gain from financial benefits as well, as a USD 619 billion revenue opportunity has been predicted for these operators in the future. This revenue opportunity is real and up for grabs by operators, but it does require business model development that elevates telecom operators beyond the connectivity play.

For further insights in this regard, and what CxOs need to do for proper facilitation of Digitalization and 5G technology, you can head over to the webinar being hosted by Eva Hedfors and Irwin van Rijssen, Head of 5G Core Program Ericsson the 20th of June.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post Digital Meets 5G; Shaping the CxO Agenda appeared first on Ronald van Loons.

Ronald van Loon

How CDOs View Data Ethics: Corporate Conscience or More Regulations

We really are in the midst of a data revolution. With a huge amount of data being generated every day, organizations in the current world are encircled left, right, and center by data and the analytic tools that are required for handling it. Leveraging this data has given companies unprecedented insights into different customer preferences and how they can cater to these needs. So, with all the emphasis on data and the capabilities it holds in the current world, should there be a question regarding data ethics?

Recently, I got a chance to attend Sapphire by SAP. At the occasion, I was asked to moderate the International Society of Chief Data Officers event along with Franz Faerber, Executive Vice President at SAP, and Michael Servaes, Executive Director, International Society of Chief Data Officers. The event was graced by some very knowledgeable attendees, who shed light on the importance of data ethics, and what the way forward is.

Speaking at the event, I talked about the different uses of data in place within the world today, and how that shapes our present and our future. Most of the companies today are not competing with their competitors anymore, but they are now up against the bar that has been set by their customers, the expectations that these customers now have of them. This bar has been set by the excellent service provided by companies such as Google and Facebook. Every touch point is important, and organizations need to realize this. Data analytics is the way forward. With all these advancements, there is a question that arises here; is the use of data going forward ethical? The role of the Chief Data Officer (CDO) here is to deliver a great customer experience by managing what they do with their data.

How to define what’s Ethical?

When it comes to data ethics, the first question that arises is about what’s ethical and what’s not. There seems to be confusion in this regard, and most organizations cannot reach a consensus on defining this. The panel that I witnessed in the CDO event had well defined answers to this question.

The first and foremost step to defining the ethical virtue of any data set is to run the “sunshine rule.” This rule basically means how you would feel about your organization if the way you used your data was out in the open. By sunshine we mean if the data was out under the sun for everyone to see. If, when thinking about the answer to the question above, you feel that you wouldn’t mind if your use of data was to go out, then you wouldn’t have any reservations in terms of ethical use of data. But, if you feel that you wouldn’t be comfortable with your data being out in the open, then you might not be using your data in the most ethical manner. This is a litmus test that has been designed for getting answers based on the truth. Give the answer based on what you truly feel, and you’ll be able to tell whether your use of data is justified or not.

Similarly, there are other tests as well that tend to cater to finding the ethical aspect of using data. These are all litmus tests. You can also imagine a scenario where your use of data is out in the newspaper, would you feel comfortable or threatened? Moreover, the panel members got on the lighter side of things and even mentioned imagining what you would feel if your significant other found out. Will you be able to stand as the same person, or would the guilt of the unethical use of data kill you? Having answers to all these questions can help you define your use of data as ethical or unethical.

The role of the Chief Data Officer

Chief Data Officers or CDOs are the leaders of the data functions in their organization. Not only this, but they play an important role in helping abide with the laws that regulation authorities have in this regard. The General Data Protection Regulation (GDPR) is one regulation that CDOs in Europe, but also in the rest of the world if you do business with European citizens, need to be well versed in. All panel members in the discussion agreed that the Chief Data Officer should keep their team in the loop at all times. It is always good to have the opinion of your team, rather than to go along with just your own opinion. If even one person in the team feels that the data is not being used in ethical ways, the CDO should be able to take the required steps to address this issue.

Until very recently, CDOs were considered the new kids on the block, and many other C-level executives didn’t rank them at the same level as themselves. However, these rising stars of the digital business have now taken the center stage on the seat towards deciding ethical standards in any organization. One of the most important parts of CDO’s is to make fine judgment calls that don’t trespass the line between trust and innovation. It is important to realize the trust organizations give CDOs and then work on it to ensure that their trust is respected.

CDOs sit at an important position, and it is their job to understand the ethical requirements of using data and eventually fulfilling those requirements. In the midst of advanced data analysis tools, it is important for CDOs to also realize the importance of giving customers the very best in terms of ethical standards, and customers deserve this trust.

Impact from Algorithms

While the world of data can be considered as impressive and transformative, there have been instances where algorithms have gone wrong. These instances have happened because of a wide variety of reasons including human biases, usage errors, technical flaws, and security vulnerabilities. For instance:

  • Many social media algorithms have gotten the wrath of viewers over how they influence public opinion. Just recently, we saw Google wrongfully accusing the views of the shooter behind the Los Angeles shooting. They later took the blame upon themselves, but these algorithms can dictate public opinion.
  • Moreover, back in 2016 during the Brexit referendum, we also saw how algorithms were blamed for being the reason behind the flash-crash that saw the pound fall by over six percent in value.
  • Moreover, investigations in the United States have also found out that algorithms in place within criminal justice systems have been biased against a certain racial group.

Best Practices

To effectively manage the ethical implications of data, CDOs should take the reins and adopt new and better approaches for building stronger foundations. There should be better algorithm management, and CDOs using data analysis tools should ensure that they take care of the ethical needs of the data they have with them. Only then would they be able to really use that data for something feasible.

We used multiple voting at the event, and came to the conclusion that data ethics should be declared as a guideline on the corporate level. More than 90 percent of the CDOs in the audience thought that way, and with growing regulations in this regard, more ethical checks on data are more of a necessity than a want.

About the Author

Ronald van Loon is, Director at Adversitement, an Advisory Board Member and Big Data & Analytics course advisor for Simplilearn.

If you would like to read more from Ronald van Loon on the possibilities of Big Data and the Internet of Things (IoT), please click “Follow” and connect on LinkedInTwitter and YouTube.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post How CDOs View Data Ethics: Corporate Conscience or More Regulations appeared first on Ronald van Loons.

 

July 13, 2018


Revolution Analytics

Because it's Friday: Language and Thought

Does the language we speak change the way we think? This TED talk by Lera Boroditsky looks at how language structures like gendered nouns, or the way directions are described, might shape they way...

...
 

July 12, 2018


Revolution Analytics

New open data sets from Microsoft Research

Microsoft has released a number of data sets produced by Microsoft Research and made them available for download at Microsoft Research Open Data. The Datasets in Microsoft Research Open Data are...

...

Simplified Analytics

How to address the reluctance for Digital Transformation?

Digital Transformation is in full swing now and adopted by almost all the industries to improve the customer experience. But not everyone is sailing smooth. In fact, a majority of Digital...

...
 

July 10, 2018


Revolution Analytics

In case you missed it: June 2018 roundup

In case you missed them, here are some articles from June of particular interest to R users. An animated visualization of global migration, created in R by Guy Abel. My take on the question, Should...

...
Ronald van Loon

Strategies for Monetizing Data: 2018 and Beyond

The data revolution is here, and it creates an investment priority for enterprises to stay competitive and drive new opportunities. One of the brightest areas is data monetization, which describes how to create economic benefits, either additional revenue streams or savings, utilizing insights provided by data resources. With B2B and B2C data needs reaching an all-time high, the monetization strategies now and into the future should be seamless for use across multiple platforms.

To get an expert view on this matter, I recently tapped Jeremy Rader, Director of Data Centric Solutions at Intel. The Opportunity for Data Monetization

Researchers have reported that the market size for big data is on the rise and is fast becoming an important distinction for organizations. This age of data means that the data culture for every organization needs to be revamped. Almost any company now has the potential to be a data company. In a research study conducted recently on big data and analytics, more than 85 percent of all respondents interviewed reported that their organizations had taken steps toward a data-driven culture. But, when asked if they had success in achieving that culture, only 37 percent replied in the affirmative.

Positioning Your Organization for Success

A key protagonist in this move toward a data culture is the Chief Data Officer (CDO), responsible for leading the figures behind data within an organization. But not every organization has a CDO, and for those organizations that do, it’s a new role with an evolving definition.

The key role of the CDO should be to take a futuristic view of an organizations’ data model that includes a data monetization strategy. This Eckerson Report includes internal recommendations for data monetization, including delivering concrete data analytics to your employees so they can prioritize, make more informed decisions and reduce costs. There are also opportunities to enrich your existing products with the use of data analytics and customer retention models, and to create a whole new product line that generates revenue by selling your data products to customers.

What is needed for Data Monetization Success?

To start, companies must be able to glean timely, in-depth insights from their data. Those insights come from the ability to access, organize and interpret the data—in effect, taking a ‘whole business’ approach to analytics.

A key focus area to help enterprises begin to align and organize around their data strategy is to get their data layer right. AI and advanced analytics workloads require massive volumes and types of datasets. To get your data ready to harness, break away from fragmented systems and older data storage models that keep your data trapped. Many organizations achieve this by implementing a modern data lake model. Then, tier your data based on its use. Your tiering strategy should include a storage model that matches your data tiers to reduce storage costs and optimize performance.

Here are some other tactics organizations should consider:

  • Establish a Clear Vision: The company’s executives should share the vision of correctly monetizing data by allocating necessary resources, including time, workforce and investment toward execution.
  • Agile Multi-Disciplinary Teams: Data monetization can be done through agile multi-disciplinary teams of data architects, product managers, application developers, analytics specialists, and marketing and sales professionals.
  • Develop a healthy, competitive, data-driven culture: Unless communicated across an organization, data remains worthless. To extract the right information and insights from structured and unstructured data, it is important to focus your efforts on cultivating a data-driven culture that empowers employees with the resources and skills they need to leverage data and obtain the right information at the right time to make more accurate decisions.
  • Ensure Easy and Secure Access to Data: For data to be monetized, it not only needs to be voluminous in size and nature, but also clean, accessible and consistent.
  • Data management & advanced analytics: A digital data management platform is essential for integration and providing solutions which are elaborate and comprehensive. A proper enterprise data management platform should contain the five service layers: engagement, integration, development, data, and modern core IT, which are the key components of every digital business. Advanced analytics provides the eventual meaning to the data through summarizations, models, calculations and categorizations. Data is valuable once it is analyzed.
  • Storage: Increased storage efficiency is critical to ensure your data is available and can be analyzed. The faster the data can be accessed while processing, the shorter the time to results, and detailed and nuanced analysis within a given response time. Intel® Optane™ DC persistent memory is a new class of memory and storage technology that better optimizes workloads by moving and maintaining larger amounts of data closer to the processor and minimizing the higher latency of fetching data from system storage.
  • Processes & delivery: A continuous development process that customizes data and analytics to your target audience needs a delivery system that provides analytics up to an advanced end-user application.

The Future of Data Strategies for Organizations Dealing With Large Data Volumes

As an example of a successful data strategy in action, the business of healthcare has an abundance of data and opportunities that can help power more accurate diagnosis and improved patient care. The stakes are high in an industry where patient outcomes are impacted by quick, early detection and treatment.

For example, Intel worked with a large health system that had an older data infrastructure with fragmented systems and data silos, which was impeding their ability to rapidly access, blend, and analyze data from multiple sources to deliver precision medicine, improve patient monitoring, and drive innovation in its’ healthcare practices. By deploying a modern data hub (Cloudera* Enterprise) running on Intel® Xeon® processors, this large health system was able to see significant results. They are using machine learning algorithms and predictive analytics at scale to anticipate and account for various patient outcomes by analyzing over 100 data points per patient per day for hundreds of thousands of patients.

There will be obstacles along the journey to get your data to a place where it can be used to answer some of your biggest challenges, but those challenges can be overcome with the right focus and investment.

The AI Revolution is Backed by Data

Intel understands that the advanced analytics and AI revolution is backed by and powered by data. The data has to be constantly maintained to achieve the ultimate potential of advanced analytics and AI. As such, Intel is focused on leading the charge for open data exchanges and initiatives, easy-to-use tools, training to broaden the talent pool, and expanded access to intelligent technology.

The data revolution will drive demand for advanced analytics and AI workloads, requiring optimized performance across compute, storage, networking and more. The recent advancements by Intel, as they usher in this paradigm shift, include the Intel® AVX-512, a workload accelerator, and the Intel® Xeon® Scalable processor based platform. Through optimized infrastructure, modern storage and data architecture, and a pathway to run complex and massively scalable analytic workloads in any environment as well as scale up and scale out with performance and agility, we can successfully enable the business of data from the edge to the cloud to the enterprise.

For more information, visit intel.com/analytics.

About the Authors:

 

Jeremy Rader, Director of Data Centric Solutions at Intel, is responsible for enabling business transformation by driving Analytics, AI and HPC solutions, while driving next generation silicon requirements. LinkedIn and Twitter.

 

 

Ronald van Loon, Director at Adversitement a data & analytics consultancy firm, and Advisory Board Member & course advisor for leading professional certification training company Simplilearn.

If you would like to read more from Ronald van Loon on the possibilities of Big Data and the Internet of Things (IoT), please click “Follow” and connect on YoutubeLinkedIn, and Twitter.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post Strategies for Monetizing Data: 2018 and Beyond appeared first on Ronald van Loons.

 

July 09, 2018


Revolution Analytics

R 3.5.1 update now available

Last week the R Core Team released the latest update to the R statistical data analysis environment, R version 3.5.1. This update (codenamed "Feather Spray" — a Peanuts reference) makes no...

...
 

July 06, 2018

Cloud Avenue Hadoop Tips

What is DIGITAL?

Very often we hear the word DIGITAL in the quarterly results of the different IT companies especially in India. The revenue from the DIGITAL business is compared with the traditional business. So, what is DIGITAL? There is no formal definition of DIGITAL, but has been loosely used by different companies as mentioned lately.

But, here is the definition of DIGITAL in an interview at MoneyControl (here) by Rostow Ravanan, Mindtree CEO and MD. This is a bit vague, but the best I could get till now. The vagueness comes from the fact that it doesn't say what BETTER is. Does anyone see something missing? I see IOT missing. Lately I had been working on IOT and would be writing my opinion on where IOT stands as of now.

Q: Digital is still a vague term in the minds of many. What does it mean for you?

A: So let me go back a little bit and tell you what we define as digital. We define digital and we put that in our factsheet, whenever we declare results every quarter.

In our definition of digital, we take one or two ways of defining it. To a business user, we definite digital from a business process perspective to say anything that allows my customer to connect to their customer better or anything that allows my customer to connect to their people better is one way of defining digital from a business process point of view.

Or if you were to look at digital from a technology definition point of view, we say it is social, mobility, analytics, cloud, and e-commerce. From a technology point of view, that is how we define digital.
 

July 03, 2018

Ronald van Loon

The Intelligent World: 5 Cases from Today

With all the talk about the smart revolution, we are finally in the smart and intelligent world, and have numerous use cases around us. While these use cases point towards the greater use of smart resources to make the world around us intelligent, there are numerous trends that can be seen coming up during the recent past. Three of these trends are:

  1. Trend 1: The digital world is finally having a bigger presence in the physical world, and we are moving towards the intelligent world.
  2. Trend 2: Digital transformation seems to be the only way towards success in the industry. It is currently the need of the hour to make correct decisions for the existing industries, and those in the future.
  3. Trend 3: The ICT infrastructure is the cornerstone for all digital platforms, as it fully supports service platforms and enables the digital transformation within any industry.

Keeping in mind these trends, one can gear up for the interesting future, as the smart world comes towards us. As part of my interests in this regard, I was recently invited by Huawei to the Huawei Booth at CEBIT, as part of their Key Opinion Leader program. For those who don’t know, Huawei is a major player in the move towards a better, more intelligent world. They have continuously invested 10 percent of their revenue from each year into driving innovation and research and development. Their R&D ranks at the 6th position of all companies across the world, and they have a large part to play in the intelligent world.

As part of the Huawei booth at CEBIT, I got to see many of the latest innovations that inspired me. Here I would point out five innovations and cases, which I experienced at CEBIT. These innovations seem to be a major part of the intelligent world, and should be thoroughly studied, to form a greater understanding of the digital world.

Smart Cities

The concept of Smart Cities has become a big part of the move towards the digital world. Smart cities were conceptualized almost half a decade ago, and we have come a long way during this time period. With many use cases being presented, one can see the use of Infrastructure as a Service and Software as Service (IaaS and SaaS) offerings in smart cities.

With the rapid development of cities across the world, it was just a matter of time before we saw a smarter method of city management roll out. Urbanization has been found to exert additional pressure on city management, governance, and industrial development. Due to this additional pressure we see economic protection, public safety, industrial and commercial subsystems, and people’s livelihood scattered in a mess. A smart city program redefines city management and uses ML (Machine Learning) and Internet of Things (IoT) technologies to augment humans in this integral part of urban management in today’s age. The use of analytics techniques helps integrate the core information about city operations.

The smart city leads to effective and collaborate services from the government, intelligent infrastructures, coordinated city emergency responses, and visualized city operations. Through this interactive convergence, traditional siloed city management transforms into a more coordinated and integrated city governance. In China alone, we see more and more cities become part of this venture, as city administrators realize the benefits in smart city management. Smart city management has helped increase law-enforcement efficiency by 150 percent in the few use cases that we have seen. The system gathers information from maps, videos, and IoT to perform visualized multi-agency operations. However, administrators need to understand the fact that smart cities are focused on pure IoT more than anything else. The role of IoT in smart cities cannot be undermined, and the sooner city planners understand this fact, the sooner they will be close to administering smart cities across the world. Simply put, having an IoT platform in the city is the first step to creating or supporting a smart city.

Smart Transportation

Click here to see the video on Youtube

With the addition of Smart Airports, we are surely moving towards smartness and intelligence in transportation. Airports are locations that have many moving parts, and with IoT, humans can increase predictive maintenance and ensure better management to improve safety and ensure comfort. A smart airport solution basically covers aspects such as visualized safety, visualized operational processes, airport IoT, and visualized services.

During the last decade, we have seen a booming increase in the global civil aviation industry. The industry has witnessed a continuous increase in passengers traveling via air and the revenue generated by the industry. Recent statistics by the International Air Transport Association (IATA) suggest that the demand for air travel will grow by over seven percent in 2018, and it is expected that the total number of passengers will also soar up high. Keeping in mind this growth, it can safely be said that civil aviation is by far one of the most rapidly growing industries across the world.

The concept behind smart airports aims to limit the downtime that aircrafts go through, and it is expected that the efficiency achieved through the implementation of the technology will help save passengers more than 2,300 hours every day. Moreover, airports can give a more feasible and time-effective service to passengers by saving them the hassle they normally go through from boarding to getting on the plane. A smart airport system leverages technologies such as Big Data, IoT, and biometric to help passengers get through all security checks by just presenting their faces. Moreover, visual surveillance would lead to better tracking without any imperative blind spots. The system may use the input from the visual surveillance to analyze sensor-triggered alarms. Alarms from sensors are often misinterpreted, which is why the system rechecks them with visual data before sending them forward. This reduces the false alarm rate by over 90 percent on average.

Smart Retail

Click to see the full interview on Twitter

The retail sector has seriously been expecting a digital revolution since the past couple of years. However, since there were numerous challenges involved in the process, retailers shied away from the innovation. The challenges relate to the downtime in deployment and the implementation of electronic price tags.

The idea behind smart retail helps to bring smart inventory management to the retail business and envisions giving retailers the feasibility they need in their business. A smart retail imitative will have the capabilities to integrate services, switches, and storage devices for supporting the mainstream visualization software.

Moreover, theft and inventory loss is one of the most concerning facts for most store-owners, and a smart retail cloud solution will keep a track of all such instances and alert the requisite agency if such an anomaly does happen. With the cloud support behind their back, smart retail providers will be able to reach out to customers on a personal basis and give them a personalized experience based on their past history and other factors.

Smart Supply Chain

Click to see the full video on Twitter

The smart supply chain endeavors to ensure optimal delivery patterns in the supply chain world of today. Supply chain is a growing industry that is embracing the changes of the digital transformation, and we will soon start seeing how much it is developing for the better. The industry disruption has happened, and the use of smart Internet resources has ensured a decrease in many of the errors that usually occur in supply chain. A smart supply chain system will take care of all the different anomalies in the process, and ensures that they are predicted through different insights, and action is taken to limit these problems.

The supply chain process should be clear of all hindrances, and through proper tracking of all goods organizations can now have a smart eye on their delivery networks. A smart supply chain network would effortlessly track all items and ensure that all goods go through the network without any hindrances.

Smart Manufacturing

The smart factory, or the concept of Industrial 4.0, connects human beings with the intelligent system of robots to create unparalleled augmentation. A fully connected factory would be able to use smart methods of production to increase their production capabilities and limit the different wastes they usually go through.

The use of smart manufacturing would generate real time analysis on the quality of the products from production to the market. The system will be able to identify parameters that have an impact on quality, and would be able to implement methods of automatic data collection. Manufacturers can use online cloud diagnosis to gather real-time and expert resources. Moreover, the machine learning engine will use the system placed inside it to quickly locate faults and present a solution for it. This efficient detection of faults can be helpful, as it saves a lot of downtime and ensures that manufacturers can focus on production rather than wasting time on errors.

After having a look at the five use cases I’ve mentioned above, and the work Huawei is doing in this regard, one can tell that the intelligent world really is upon us. And, propelling this intelligent world of smart networks forward is none other than Huawei.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post The Intelligent World: 5 Cases from Today appeared first on Ronald van Loons.

Ronald van Loon

Telecom: Digital Transformation in the Sales and Network Distribution

The digital transformation is upon us, and telecom operators seem to be the ones directly under the impact of it. The advent of new data and technological needs has meant that we now need to give nothing less than the best to our customers. As part of this assurance of the best customer service, telecom operators need to up their game when it comes to sales and network distribution.

Due to technology and the Internet, we now have new and refreshed distribution channels. These distribution challenges are centered on data driven insights, and telecom operators have to work on them to get their distribution game right. Operators also have to work on multi-channel networks that deliver the goods. Due to this digital transformation, operators now have to naturally make alterations in their sales and distribution strategy. There is a need for a paradigm shift in this regard, and operators have to endeavor towards it, rather than waiting for the shift to happen.

Challenges while Implementing an Effective Sales and Distribution Strategy

With all the talk about the need for telecom operators to change their sales and network distribution techniques, this change won’t be as easy as it sounds. Telecom operators have to realize the need for complete transparency and ensuring that there is no flaw in the process. There are certain complications that telecom operators will face in this regard. Here we look at some of these complications:

  • High value transaction being executed through credit sales: One of the biggest challenges impeding the way of telecom operators towards the revolution in sales and distribution is the credit challenge. Operators will have to realize that they will incur big transactions through credit sales. They have to take care of their liquidity and make a future plan based on this concern for liquidity. They cannot slack off here, as the wrong plan might lead them to disaster in their cash stream.
  • Payouts through commissions: Their employee payouts will mostly be focused on commissions, as employees will try to achieve customer retention and complex sales targets. The criteria for employee evaluation cannot be the same, as going digital requires a change in the perception.
  • Unrealized Inventory: Unrealized inventory will create another hindrance towards efficiency. Operators should be able to realize inventory, and not hide efficiencies under the sea of inventory. Efficiency is important here, and should not be ignored.

Defining a Successful Strategy

While one cannot point out any sure shot strategy towards success in the implementation of better sales and distribution methods, the basic idea behind every strategy should be to sell the right product to the right customer at the right time, through the selection of all appropriate channels. In addition to this, modernizing all existing sales is something that needs to be done. A success strategy would select areas where this method can properly be implemented.

Shrinking Partner Hierarchies 

Telecom operators are currently involved in numerous partnerships with various suppliers. These partnerships require suppliers to fulfill requests for all new services, rather than just forwarding them. With the supplier becoming a partner, their role ultimately changes. Partners now have to receive devices from third party providers, bundle these devices, and upsell them. With this idea established, one cannot go without comprehending the disruption in the supply chain that is about to happen. A product-specific relationship is to be established between resellers and retailers, and the idea is to ensure that any supplier can provide any product to any retailer.

Blurring of Channel Boundaries 

Each sales channel, in traditional methods of selling, had different partners and suppliers. These sales channels were used to push forward products that overlapped, but were different from each other. However, this is about to change. In this era of digitalization, non-direct channels have to cater to new demands by playing a dual role.

This dual role is required, because customers are expected to interact with operators through channels that they have chosen themselves. Operators must ensure product availability at these channels, since they will now be functioning as device collection points and as extended customer service endpoints. Simply put, operators will have to become creative and make non-direct channels function more like direct channels.

Management of Territory and Sales KPIs 

Since digitization has brought a much-needed change in the both partner and channel structures, it is about time that sales measurement techniques are overhauled. Measuring sales focused on territorial data would soon become outdated, and leave organizations wondering. Sales and distribution teams will have to define newer boundaries. Systems that do not account for the sales achieved through digital channels will not meet future requirements and expectations. There is a need for defining newer logical territorial boundaries that overlay the boundaries for physical territories. In some cases, operators will have to completely eliminate the requirement for traditional methods of sales measurement through territory.

A Future Roadmap

With digitization expected to signal towards a paradigm shift in how sales and distribution channels for telecom operators work, there needs to be a unique approach for making your sales system highly configurable. Some potential features of this approach include:

Go Mobile 

All operators utilizing a solution in sales and distribution will go towards a mobile interface as the de-facto option. The mobile interface will, however, have to account for all the active players in the market. These active players in the value chain include distributors, operators, and end subscribers. All of these players need to be wary of what is coming, and they should work along these lines in the future.

Analytics and Offer Management 

Although analytics have found their way through most enterprises, sales and distribution systems are still limited when it comes to use cases. The use of analytics in sales and distribution is ideal, as analytics offer multiple benefits. Operators will have to gather data and perform analytics to get feasible analysis out of it. Once they have actionable insights, they will be able to gather necessary data for promoting new products through distributors. While the use of analytics in sales and distribution might come across currently as too early for its time, there is a massive need for understanding how it can revolutionize the shift towards the digital world. Operators that use these methods will stand above the competition when it comes to understanding sales patterns and distribution efficiency.

Simplify Management of Sales and Channel 

It is necessary for operators to simplify the management of sales funnels and channels. As we mentioned above, non-direct channels may stand above the rest, but there is still some need for simplification. Operators can opt for an API gateway, which may reduce the hassle involved with sales channels. With digitization, the sales ecosystem may become even more complex, which is why a robust infrastructure, provided through an API gateway, is a necessity.

If you want to read more about the digital transformation in sales and distribution, and what telecom operators should do about it, you can go and read the online e-book by MahindraComviva here.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post Telecom: Digital Transformation in the Sales and Network Distribution appeared first on Ronald van Loons.

 

June 29, 2018


Revolution Analytics

Because it's Friday: Wavy Lines Illusion

Another fine illusion: in this one, the pairs of horizontal lines are all smooth sine curves, despite the appearance of the jagged zig-zags: It's really hard for me at least to tell that the zig-zags...

...

Revolution Analytics

Global Migration, animated with R

The animation below, by Shanghai University professor Guy Abel, shows migration within and between regions of the world from 1960 to 2015. The data and the methodology behind the chart is described...

...
Ronald van Loon

The Future of AI is Here

For the last five years, we’ve been discussing the future of AI with many organizations, including Intel, at the forefront of that conversation. However, while ensuring the world gets to experience new heights and unparalleled technologies, organizations failed to notice that the future is upon us. This realization dawned on me when I attended AI Devcon in San Francisco as an Intel Partner. The conference was hosted by several opinion leaders, such as  Naveen Rao, the VP and GM of artificial intelligence at Intel. The conference helped me gain insight into the impact of our efforts and brought me to the realization that it is time other organizations jumped on the AI bandwagon initiated by Intel.

Why Do You Need to Start Using AI Now

There are multiple reasons you should start leveraging AI. The market presents a lot of potential for the future and set unprecedented records; that future seems bright. The current market growth potential for AI is high, and the market which currently stands at $3 Trillion is expected to grow to over $8 Trillion in the next five years. That is roughly a 200 percent growth over that period.

Besides the increasing market growth for AI, there is also a need to increase customer experiences. Customers now want the best experience possible and organizations that can offer those experiences will come out on top. Thus, if you want to ensure the right experience for your customers, I can’t stress enough the importance of jumping onto the AI bandwagon.

Examples of AI Implementation

AI is currently being implemented across a number of industries, and there are numerous use cases that set a standard when leveraging AI. Here we look at some of those use cases, and how organizations can pull from them for their own industries.

Improving Patient Outcomes

Medical data is extremely difficult to measure and analyze, which is why only a data science setup with the capabilities to host this complicated data can actually garner presentable insight. There are numerous cases of Intel working with medical institutions to improve patient outcomes through the collection and analysis of data. By gaining insight that was previously not available, medical experts can now use AI to give patients the right treatments at the right time. This helps improve patient outcome and increases the credibility of data science and AI tools.

Image Rendering in Filmmaking

The use of AI in image rendering for filmmaking improves user experience and ensures that everyone gets to witness a flawless experience while watching a film. AI can be used in filmmaking to increase the graphic representation of different living animals. The data from the movement and stimulation of animals is perfectly represented inside the film to create an honest representation of the movements made by living creatures.

Real-time AI Music

Real-time AI music is now a reality and Intel’s Movidius sits at the forefront of such advances. The technology has been credited with using set responses to add value to music and create a rhythmic tone. The model gathers insights and creates responses based on the frequencies of the content.

Machine Learning at AWS

Amazon has been leveraging machine learning to provide customers with suggestions and better understand their needs. Amazon is also using machine learning to create innovations in devices such as Alexa and Amazon Go. Amazon Sagemaker, which is at the forefront of Amazon’s machine learning initiative, brings machine learning to the cloud to benefit developers and enterprises.

Use of AI by Ferrari

The use of AI in a Ferrari is geared towards helping achieve the following functions:

  • Helping drivers achieve faster times in race circuits.
  • Helping engineers pioneer desired responses from the engine of the car.

These insights have been garnered through intelligent data sets achieved through drone technology.

What is the Basis of AI Machine Learning?

Machine learning is based on four types of learning:

  • Supervised Learning: What we see in the world today is supervised learning, where machines are supervised and fed data tools required for garnering actionable insight.
  • Transfer Learning: The transfer of knowledge you get from one insight into another data set. Transfer learning gives organizations the leverage they need to make machines learn from examples.
  • Unsupervised Learning: Learning without the presence of able data. This means to learn without the presence of specific variables. Unsupervised learning has transformed the concept of machine learning.
  • Reinforcement Learning: Reinforcement learning provides an infinite amount of experience and data. Reinforcement learning gathers actionable insight from that data. Model based reinforcement learning spurs from this method.

AI and Ethics

There will be a big discussion on ethics when AI starts making its own decisions. Whether these decisions comply with the human ethical standards that we currently follow is something that remains to be seen. We can take the example of a self-driven car in this use case. The car would have to make decisions, such as colliding into a passerby or not colliding into a pedestrian or going into a brick wall nearby. What’s interesting is that these decisions will be taken in real-time, and how AI gets to pull this ethical implication off is something that defines our future.

The growth of AI during the next 50 years can be envisioned. All that is needed to propel this growth forward is a solid infrastructure, software and facilities. The infrastructure will be provided by different hardware tools and the community with the assistance of developers. Once these developers have the necessary infrastructure, we will see a broader implementation of AI across the globe.

Want to learn more about AI watch the keynotes at Intel AI Devcon, click here.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post The Future of AI is Here appeared first on Ronald van Loons.

 

June 28, 2018


Revolution Analytics

Should you learn R or Python for data science?

One of the most common questions I get asked is, "Should I learn R or Python?". My general response is: it's up to you! Both are popular open source data platforms with active, growing communities;...

...
InData Labs

Using Multilevel Modeling for Secure and Effective Dam Operation and Floods Prediction

Floods are among Earth’s most common and most destructive natural hazards. Floods occur when water overflows the land that is normally dry. Excessive rains can also lead to dams rapture or simple water overflow, which causes unwanted consequences for downstream settlements and unoptimized electricity production. Most floods take hours and even days to develop, giving...

Запись Using Multilevel Modeling for Secure and Effective Dam Operation and Floods Prediction впервые появилась InData Labs.

Solaimurugan V.

Big Data / Data Analytics Job

(adsbygoogle = window.adsbygoogle || []).push({ google_ad_client: "ca-pub-2788927527603741", enable_page_level_ads: true }); Data Analytics job in UK. Big data analytics Jobs body{ margin: 0 ; padding: 0; } .red{color:red;} .green{color:green;word-wrap: break-word;} //media screen having max width of 500px @media only screen and (max-width:500px){ } @media screen
 

June 27, 2018


Revolution Analytics

The Financial Times and BBC use R for publication graphics

While graphics guru Edward Tufte recently claimed that "R coders and users just can't do words on graphics and typography" and need additonal tools to make graphics that aren't "clunky", data...

...

Simplified Analytics

How Digital Technologies are booming the Real Estate

Digital disruptions are impacting all the industries and pushing organizations to change or die.  The residential real estate industry was built upon personal relations and contacts. Knowing...

...

Datameer

Cloud Build or Buy? Answers for the Enterprise

In today’s fast-paced world, organizations must constantly adapt and innovate to maintain their competitive advantage. Cloud computing has revolutionized the technological landscape, allowing...

...
Solaimurugan V.

3 Best Machine Learning Framework for design, experiment and deployment

Open source Machine Learning platform body{ margin: 0 ; padding: 0; } .font2{ font-size: 13px; #font-family:arial,sans-serif #font-family:"Comic Sans MS"; } .round{ width:8%; height: 6%; display: inline-block; border-radius: 50%; } .font3{ font-size: 15px; font-family: monospace; #text-shadow:
 

June 26, 2018

Knoyd Blog

Exploring the Neighbourhoods of Vienna with Open data

In this article, we look at 'the built environment', which Wikipedia describes as "human-made surroundings that provide the base for human activity, containing everything from parks, public transport or hospitals to coffees, bars, and restaurants." Recently, we have seen the expansion of smart cities as well as data-driven approaches to urbanism. This means that people are trying to use data to better understand how they live, how different neighbourhoods are built, and what the similarities between them are. One way to do it is to explore the neighbourhoods through open data. 

 

Data Sources Description

For this blog post, we have chosen the city of Vienna, Austria for the following reasons:

  • Size of the city
  • Availability of open datasets
  • Soft spot for the city because I lived there ;-)

If you have visited Vienna before, you might have seen the famous St. Stephen's Cathedral, tried the traditional Wiener Schnitzel, but you also know that the city is divided into 23 districts. These districts play an important part for Viennese people. Actually to such extent that one of the first questions locals usually ask you is: "Which district do you live in?" The goal of this blog post is to analyse these districts (neighbourhoods) using public data sources to find differences and similarities among them. We have used three different types of data sources: Vienna open data, scraping of various real-estate agencies websites using BeautifulSoup, and the Google Places API.

 

Approach

We decided to use a clustering algorithm so that we can assign the same cluster to similar neighbourhoods. For this, we have used information pulled from the Google Places API. We gathered the number of various places per district, including doctors, restaurants, bars, coffee shops, parks, museums, galleries, and even beauty and hair care salons. With regard to the number and type of these places, the two closest districts are 4 (Wieden) and 6 (Mariahilf), with only 16-point distance, while the distance between the two most distinct neighbourhoods (22 (Donaustadt) and 5 (Margareten)) was 180.

We excluded any demographic or economic indicators from the clustering but some interesting relationships were found between the clusters. Furthermore, we have scraped the actual prices of houses and apartments in Vienna from a couple of real-estate websites to further enhance the profiles of our clusters. The distribution of apartment prices can be seen in the following charts:

 Histogram of sizes of apartments that are currently for sale in Vienna

Histogram of sizes of apartments that are currently for sale in Vienna

 Histogram of prices per m2 of apartments that are currently for sale in Vienna

Histogram of prices per m2 of apartments that are currently for sale in Vienna

 

The Solution

Using the places pulled from Google API, we have clustered the neighborhoods into 5 different clusters.

Cluster number  Districts
--------------------------------------------------------------
Cluster 1       1010
Cluster 2       1020 - 1130
Cluster 3       1030 - 1100 - 1110 - 1140 - 1190
Cluster 4       1040 - 1050 - 1060 - 1070 - 1080 -1090 - 1120 - 
                1150 - 1160 - 1170 - 1180 - 1200
Cluster 5       1210 - 1220 -1230

Cluster 1 consists only of the district number 1 (Innere Stadt), because is very far away from any other district. This was further proven by profiling, using demographic data and prices of apartments. This district is home to some of Vienna's biggest attractions such as St. Stephans Cathedral, Hofburg Palace, and the famous Opera House.

 Comparison of District Innere Stadt against average values in Vienna

Comparison of District Innere Stadt against average values in Vienna

We can see that District 1 is very far from the average in various categories, especially the apartment prices, which are almost 3-times higher than the overall Viennese average — despite the fact that the royal apartments in the Hofburg Palace aren't currently for sale. Also we can see, that people who live here are older on average, which leads to significantly fewer families compared to other parts of the city.

As we can see in other clusters, it is often the case that similarity doesn't always coincide with geographical distance of the neighbourhoods. For example, in cluster 3, there are districts that are far from each other on the map, like 11 (Simmering) and 14 (Penzing). However, they are pretty close in terms of demographics and apartment prices. Be also aware that the clustering algorithm has never seen these latter attributes. The clustering was done purely based on the places pulled from Google API.

 The comparison of 2 districts in the same cluster

The comparison of 2 districts in the same cluster

Another interesting finding was, that two districts which are separated from the rest of the city by the Danube River are together in one cluster. However, they are very similar to district no. 23, which is on the opposite side of Vienna. 

 

Summary

Nowadays, there is a lot of relevant information publicly available either as an open data initiative, APIs, or simply as content on various websites. In order to keep up with current trends, it is crucial for any organisation, whether it is a public or a private one, to use this information for their own or people's benefit. If you are currently thinking about tackling a similar problem or struggling with processing a lot of information, don't hesitate to contact us.

Get in touch

Permalink

 

June 25, 2018

Solaimurugan V.

Big Data / Data Analytics Jobs

(adsbygoogle = window.adsbygoogle || []).push({ google_ad_client: "ca-pub-2788927527603741", enable_page_level_ads: true }); Data Analytics job in UK. Big data analytics Jobs body{ margin: 0 ; padding: 0; } .red{color:red;} .green{color:green;word-wrap: break-word;} //media screen having max width of 500px @media only screen and (max-width:500px){ } @media screen
 

June 22, 2018


Revolution Analytics

Because it's Friday: The lioness sleeps tonight

Handlers for the lion enclosure at San Diego Zoo have developed a novel way to provide stimulation for their big cats: let them play tug-of-war with people outside. People plural that is — it turns...

...
Big Data University

Read and Write CSV Files in Python Directly From the Cloud

Every data scientist I know spends a lot of time handling data that originates in CSV files. You can quickly end up with a mess of CSV files located in your Documents, Downloads, Desktop, and other random folders on your hard drive.

I greatly simplified my workflow the moment I started organizing all my CSV files in my Cloud account. Now I always know where my files are and I can read them directly from the Cloud using JupyterLab (the new Jupyter UI) or my Python scripts.

This article will teach you how to read your CSV files hosted on the Cloud in Python as well as how to write files to that same Cloud account.

I’ll use IBM Cloud Object Storage, an affordable, reliable, and secure Cloud storage solution. (Since I work at IBM, I’ll also let you in on a secret of how to get 10 Terabytes for a whole year, entirely for free.) This article will help you get started with IBM Cloud Object Storage and make the most of this offer. It is composed of three parts:

  1. How to use IBM Cloud Object Storage to store your files;
  2. Reading CSV files in Python from Object Storage;
  3. Writing CSV files to Object Storage (also in Python of course).

The best way to follow along with this article is to go through the accompanying Jupyter notebook either on Cognitive Class Labs (our free JupyterLab Cloud environment) or downloading the notebook from GitHub and running it yourself. If you opt for Cognitive Class Labs, once you sign in, you will able to select the IBM Cloud Object Storage Tutorial as shown in the image below.

IBM Cloud Object Storage Tutorial

 

What is Object Storage and why should you use it?

The “Storage” part of object storage is pretty straightforward, but what exactly is an object and why would you want to store one? An object is basically any conceivable data. It could be a text file, a song, or a picture. For the purposes of this tutorial, our objects will all be CSV files.

Unlike a typical filesystem (like the one used by the device you’re reading this article on) where files are grouped in hierarchies of directories/folders, object storage has a flat structure. All objects are stored in groups called buckets. This structure allows for better performance, massive scalability, and cost-effectiveness.

By the end of this article, you will know how to store your files on IBM Cloud Object Storage and easily access them using Python.

 

Provisioning an Object Storage Instance on IBM Cloud

Sign up or log in with your IBM Cloud account here (it’s free) to begin provisioning your Object Storage instance. Feel free to use the Lite plan, which is free and allows you to store up to 25 GB per month. You can customize the Service Name if you wish, or just leave it as the default. You can also leave the resource group to the default. Resource groups are useful to organize your resources on IBM Cloud, particularly when you have many of them running. When you’re ready, click the Create button to finish provisioning your Object Storage instance.

Creating an Object Storage instance

Working with Buckets

Since you just created the instance, you’ll now be presented with options to create a bucket. You can always find your Object Storage instance by selecting it from your IBM Cloud Dashboard.

There’s a limit of 100 buckets per Object Storage instance, but each bucket can hold billions of objects. In practice, how many buckets you need will be dictated by your availability and resilience needs.

For the purposes of this tutorial, a single bucket will do just fine.

Creating your First Bucket

Click the Create Bucket button and you’ll be shown a window like the one below, where you can customize some details of your Bucket. All these options may seem overwhelming at the moment, but don’t worry, we’ll explain them in a moment. They are part of what makes this service so customizable, should you have the need later on.

Creating an Object Storage bucket

If you don’t care about the nuances of bucket configuration, you can type in any unique name you like and press the Create button, leaving all other options to their defaults. You can then skip to the Putting Objects in Buckets section below. If you would like to learn about what these options mean, read on.

Configuring your bucket

Resiliency Options

Resiliency OptionDescription
Characteristics
Cross RegionYour data is stored across three geographic regions within your selected locationHigh availability and very high durability
RegionalYour data is stored across three different data centers within a single geographic regionHigh availability and durability, very low latency for regional users
Single Data CenterYour data is stored across multiple devices within a single data centerData locality

Storage Class Options

Frequency of Data AccessIBM Cloud Object Storage Class
ContinualStandard
Weekly or monthlyVault
Less than once a monthCold Vault
UnpredictableFlex

Feel free to experiment with different configurations, but I recommend choosing “Standard” for your storage class for this tutorial’s purposes. Any resilience option will do.

After you’ve created your bucket, store the name of the bucket into the Python variable below (replace cc-tutorial with the name of your bucket) either in your Jupyter notebook or a Python script.

Creating Service Credentials

To access your IBM Cloud Object Storage instance from anywhere other than the web interface, you will need to create credentials. Click the New credential button under the Service credentials section to get started.

In the next window, select Manager as your role, and add {"HMAC":true} to the Add Inline Configuration Parameters (Optional) field. You can leave all other fields as their defaults and click the Add button to continue.

You’ll now be able to click on View credentials to obtain the JSON object containing the credentials you just created. You’ll want to store everything you see in a credentials variable like the one below (obviously, replace the placeholder values with your own). Take special note of your access_key_id and secret_access_key which you will need for the Cyberduck section below.

Note: If you’re following along within a notebook be careful not to share this notebook after adding your credentials!

 

Putting Objects in Buckets

There are many ways to add objects to your bucket, but we’ll start by taking a look at two simple ways: the IBM Cloud web interface and Cyberduck.

IBM Cloud Web Interface

You can add a CSV file of your choice to your newly created bucket through the web interface by either clicking the Add objects button, or dragging and dropping your CSV file into the IBM Cloud window.

If you don’t have an interesting CSV file handy, I recommend downloading FiveThirtyEight’s 2018 World Cup predictions.

Cyberduck

Cyberduck is a free cloud storage browser for Mac OS and Windows. It allows you to easily manage all of the files in all of your object storage instances. After downloading, installing, and starting Cyberduck, create a new bookmark by pressing +Shift+B on Mac OS or Ctrl+Shift+B on Windows. A window will pop up with some bookmark configuration options. Select the Amazon S3 option from the dropdown and fill in the form as follows:

  • Nickname: enter anything you like.
  • Server: enter your service endpoint. You can choose any public endpoint here. For your convenience, I recommend one of these:
    • s3-api.us-geo.objectstorage.softlayer.net (If you live in the Americas)
    • s3.eu-geo.objectstorage.softlayer.net (if you live in Europe)
    • s3.ap-geo.objectstorage.softlayer.net (if you live in Asia)
  • Access Key ID: enter the access_key_id you created above in the Creating Service Credentials section.

Close the window and double-click on your newly created bookmark. You will be asked to log in. Enter the secret_access_key_id you created above in the Creating Service Credentials section and click Login.

You should now see a file browser pane with the bucket you created in the Working with Buckets section. If you added a file in the previous step, you should also be able to expand your bucket to view the file. Using the action dropdown or the context menu (right-click on Windows, control-click on Mac OS).

You can add files to your buckets by dragging and dropping them onto this window.

Whether you use the IBM Cloud web interface or Cyberduck, assign the name of the CSV file you upload to the variable filename below so that you can easily refer to it later.

 

Reading CSV files from Object Storage with Cyberduck

Once you have successfully accessed an object storage instance in Cyberduck using the above steps, you can download files by double-clicking them in Cyberduck’s file browser. You can also generate links to your files by selecting the Open/Copy Link URL option.

By default your files are not publicly accessible, so selecting a URL that is not pre-signed will not allow the file to be downloaded. Pre-signed URLs do allow for files to be downloaded, but the link will eventually expire. If you want a permanently available public link to one of your files, you can select the Info option for that file and add READ permissions for Everyone under the permissions section.

 

After changing this setting you can share the URL (without pre-signing) and anyone with the link will be able to download it, either by opening the link in their web browser, or by using a tool like wget from within your Jupyter notebook, e.g.

Reading CSV files from Object Storage using Python

The recommended way to access IBM Cloud Object Storage with Python is to use the ibm_boto3 library, which we’ll import below.

The primary way to interact with IBM Cloud Object Storage through ibm_boto3 is by using an ibm_boto3.resource object. This resource-based interface abstracts away the low-level REST interface between you and your Object Storage instance.

Run the cell below to create a resource Python object using the IBM Cloud Object Storage credentials you filled in above.

After creating a resource object, we can easily access any of our Cloud objects by specifying a bucket name and a key (in our case the key is a filename) to our resource.Object method and calling the get method on the result. In order to get the object into a useful format, we’ll do some processing to turn it into a pandas dataframe.

 

We’ll make this into a function so we can easily use it later:

Adding files to IBM Cloud Object Storage with Python

IBM Cloud Object Storage’s web interface makes it easy to add new objects to your buckets, but at some point you will probably want to handle creating objects through Python programmatically. The put_object method allows you to do this.

In order to use it you will need:

  1. The name of the bucket you want to add the object to;
  2. A unique name (Key) for the new object;
  3. A bytes-like object, which you can get from:
    • urllib‘s request.urlopen(...).read() method, e.g.
      urllib.request.urlopen('https://example.com/my-csv-file.csv').read()
    • Python’s built-in open method in binary mode, e.g.
      open('myfile.csv', 'rb')

To demonstrate, let’s add another CSV file to our bucket. This time we’ll use FiveThirtyEight’s airline safety dataset.

You can now easily access your newly created object using the function we defined above in the Reading from Object Storage using Python section.

Get 10 Terabytes of IBM Cloud Object Storage for free

You now know how to read from and write to IBM Cloud Object Storage using Python! Well done. The ability to pragmatically read and write files to the Cloud will be quite handy when working from scripts and Jupyter notebooks.

If you build applications or do data science, we also have a great offer for you. You can apply to become an IBM Partner at no cost to you and receive 10 Terabytes of space to play and build applications with.

You can sign up by simply filling the embedded form below. If you are unable to fill the form, you can click here to open the form in a new window.

Just make sure that you apply with a business email (even your own domain name if you are a freelancer) as free email accounts like Gmail, Hotmail, and Yahoo are automatically rejected.


Revolution Analytics

A guide to working with character data in R

R is primarily a language for working with numbers, but we often need to work with text as well. Whether it's formatting text for reports, or analyzing natural language data, R provides a number of...

...
 

June 21, 2018


Datameer

DataOps: New Term, Similar Concepts

Componentization, containers and the cloud have all converged to usher in a new era focused on “Ops.” It started with DevOps, which according to Wikipedia is defined as: “a software engineering...

...

Revolution Analytics

AI, Machine Learning and Data Science Roundup: June 2018

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications I've...

...
 

June 20, 2018


Revolution Analytics

PYPL Language Rankings: Python ranks #1, R at #7 in popularity

The new PYPL Popularity of Programming Languages (June 2018) index ranks Python at #1 and R at #7. Like the similar TIOBE language index, the PYPL index uses Google search activity to rank language...

...

Curt Monash

Brittleness, Murphy’s Law, and single-impetus failures

In my initial post on brittleness I suggested that a typical process is: Build something brittle. Strengthen it over time. In many engineering scenarios, a fuller description could be: Design...

...

Curt Monash

Brittleness and incremental improvement

Every system — computer or otherwise — needs to deal with possibilities of damage or error. If it does this well, it may be regarded as “robust”, “mature(d),...

...
Making Data Meaningful

MicroStrategy – Have you looked at THEM lately?

If you haven’t done so lately, it’s time to take another look at MicroStrategy.  They have done a great job in updating their offerings to match what is currently hot in the marketplace.  They couple eloquently form and function together to please customers and technicians.  You are sure to find interesting and thought provoking content within their product offerings.  And you might even find things that have personal application as well as business application.

Here are the latest focus points in their strategy:

MicroStrategy Cloud Intelligence

One of the pillars MicroStrategy is focusing on is Cloud Intelligence.  The details about the structure and function of the cloud are easy to understand.  It is a seamless fit into the Microstrategy BI environment.  After an initial perusal of the offering, it is easy to see the power, flexibility, and security of cloud computing, and how one is able to utilize it to drive their business decisions and adoption within organizations.  MicroStrategy has gone to great lengths to highlight the advantages, the steps necessary to setup and deploy, and as a result benefit from the MicroStrategy Cloud.  I believe this is a special niche that those who can visualize how to ramp up business intelligence projects without a lot of the normal overhead of software/hardware procurement as part of projects.  One is left to focus on what they do best, and leverage an optimized platform as part of the overall deliverable.

MicroStrategy Mobile Intelligence

Microstrategy has bet the business to emphasize the importance of mobile intelligence.  They believe that it will overtake the traditional web browser based intelligence that is prevalent today.  MicroStrategy focuses on educating the business and developer community about the value of the MicroStrategy Mobile platform.  It is easy to gain access to learn how to use the MicroStrategy platform to design, build out, maintain, support, and customize visually enticing apps for multiple output devices (iPad, iPhone), while leveraging the enterprise-caliber features of the MicroStrategy BI platform. This is achieved by implementing the metadata layer that governs all content.  Highlighting such functionality, it helps to show that MicroStrategy is clearly the market leader within this pillar and there were customer stories to back up this claim.

MicroStrategy Social Intelligence

This is the most unusually interesting pillar due to the cutting edge nature of it.  MicroStrategy Social Intelligence solutionsis designed for both commercial customers and the consumer in the marketplace.  MicroStrategy has built a bridge between the two that is compelling and an opportunity for those that have the courage to leverage it.  MicroStrategy latest offerings that enable in-depth analysis of the Facebook fan base.  They also focused on how to apply the research in the wealth of information available at Facebook to deliver very effective marketing campaigns, which basically makes the older style CRM systems obsolete.  MicroStrategy walked through the steps and their products that help make this happen, which turn the promise of social media content into real business opportunities.  Once engaged, a loyal fan-base turns into revenue… and companies that understand this and who those customers are, they will achieve a competitive advantage.

MicroStrategy Big Data

Big Data is here now.  MicroStrategy has methods and technology that help clients deal with the extreme data volumes.  The point was made that companies need to have the ability to use very large databases and data sets to make intelligent business decisions to drive growth and gain competitive advantages.  Often interesting information is lurking in the details and MicroStrategy provides a method to make sense of it.  MicroStrategy also offer features such as improved self-service that reduces the reliance on IT, when it comes to navigating the business intelligence architecture.  There is even the possibility to connect MicroStrategy to Hadoop and begin to analyze web logs in a very easy to consume fashion.  In addition, MicroStrategy focused on high performance across the entire platform to eliminate latency issues and meet performance goals.

The time is now to take a fresh look at Microstrategy.  They are a big time player in the tools space to enable Business Intelligence.  You won’t regret it.

The post MicroStrategy – Have you looked at THEM lately? appeared first on Making Data Meaningful.

 

June 19, 2018

Ronald van Loon

5 Keys to Creating a Data Driven Culture

Businesses across the world are embarking towards a digital data transformation. The transformation has really sped up during the last decade or two, and progress towards it has become even faster. Now we stand at an important moment in time, where digital technologies are massively influencing the way most customers interact with their brands. Considering this change in preferences and style of communication, it is now up to all businesses to realize the potential for development and what they need to know for being part of the wave towards the future. Businesses need to prepare themselves for the modern world of technology, and for that, they need to step into the digital world. A recent researchstudy by Gartner has revealed that almost all the businesses have started ranking digital business growth as one of their top 10 business objectives for the future.

What is causing this Disruption?

With business reconstructing their entire model to accommodate the need for digital change, one does think about what is causing this disruption.

  • The need for a digital change starts with data. Data has become the need of the hour, and to perfectly manage and extract it, organizations need to go where the customers are: digital. Data is being generated by customers on the digital world, and organizations are willing to incorporate this digital change, in a bid to get hold of this data. IoT devices are playing an important role in data generation. Most IoT devices, coupled with the smartphones using them, curate data that is of great importance for all organizations.
  • Customers are not the only ones generating this data. From smart city technologies such as connected cars, trains, and video surveillance, to businesses themselves, data is coming in at a rate of knots. The digital interactions that every business has with their customers is one of the major sources of data, and businesses often ponder over how they could use these data sources to reach meaningful insights that help them in real time.

Challenges Hindering Digital Transformation

The digital transformation process may be the need of the hour, but this doesn’t mean that it is absent of any challenges. The process is currently hindered by numerous challenges, and we will be talking about them briefly here.

  1. Data: While data is leading the digital transformation, it also presents one of the biggest challenges to organizations looking to jump on the bandwagon. Data coming from multiple sources can create a hassle for an organization when it comes to extracting meaningful insights from this data. More than 60 percent of all data that is being extracted by organizations goes to waste, since it is unused by analytics. While initially the problem laid in the collection and storage of this data, but now the implication lays in how to get meaningful insights from all the data that you have collected.
  2. Legacy Systems: Businesses that have legacy systems in place within their organization might find jumping to the digital transformation hard. Organizations, such as banks, have paper records more than 50 years old, so going digital for them isn’t going to be a walk in the park.
  3. Data Security and Privacy: A data infrastructure can often be at risk from external attacks that may endanger the security and privacy of your data. The data you have is a collection of information related to your organization, your Business to Consumer (B2C) clients, your Business to Business (B2B) clients, and most importantly personal information related to your organization. Is going digital a risk work taking? How effective can safety procedures be in safeguarding from a privacy attack? And, what is the course of action in the case of a data breach? These are some of the questions being asked by curious organizations.
  4. Lack of Information Pertaining to the Transformation: Believe it or not, but most organizations are still not educated about the perks of the digital transformation, and how they can properly implement it. The digital transformation is an amalgamation of numerous steps, which is why you need to take every aspect into consideration.

How to Overcome the Challenges to Digital Transformation, and Develop a Proven Strategy

Having talked about the complications present in being part of the digital transformation, we will now take a look at what one can do to develop a proven strategy. The five keys to creating a data driven culture in your organization are:

Have a Clear Vision

The first aspect of setting a data driven strategy is to have a concise vision that leads to successful data-centricity across all facets of your organization. This begins with the better understanding of data analysis patterns, since they are responsible for finding meaning out of your data. The most successful organizations take the help of data analysis tools to drive their digital efforts forward. A data analytics environment focused on Artificial Intelligence (AI) and Machine Learning (ML) will eventually help extract insights from data.

Establish Data Accessibility

The main motive behind implementing a data driven ecosystem within your organization is to ensure that all the data you gather and collect from different sources is easily accessible to your entire workforce. Improved access for everyone will create a positive environment of collaboration, where your entire workforce will work towards achieving the goods of the digital transformation. This will ensure better co-ordination and advanced analysis. To make data accessible for all, you can have it stored on the cloud. The cloud is accessible to all, and would ensure that the data is safeguarded through different privacy features.

Interdepartmental Collaboration

People, not tools, act as the catalyst of change inside an organization. So your change will be incomplete without the incorporation of diverse teams with unique and accurate skill sets. Making use of multidisciplinary teams that are comprised of data scientists, engineers, developers, and analysts will help create a notion of positive change.

Well Maintained Data

Data maintenance is another very important aspect of the data revolution that shouldn’t be ignored. The data needs to be clean, and should be readily available for analysis when the need arises. Data cleansing and enrichment are practices that should be incorporated for proper management and maintenance of your data. You can also incorporate all of your data input in one location to ensure that it is easily accessible, and that management is easier.

Reward System for Encouraging Internal Competition

Encouraging some competition in your team will help bring forth the real spirit of the data driven culture. Your team members can be offered numerous rewards and other incentives to motivate them towards being involved in the data culture. If the company’s objective is to penetrate into new markets, then you can set incentives on how well your team is performing in this regard. Keep your data team motivated, and you may well reap the rewards.

Once you have implemented these five key factors for creating a data driven culture, you can start reaping the rewards of the digital data transformation. Remember that extracting meaningful insight is one of the major parts of this transformation, and you should be focused more on this than anything else. A data driven culture will help you extract insights from data, while being truly part of the digital transformation.

For more insights on why you need a data-driven culture, you can register yourself for the MicroStrategy Webinar.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post 5 Keys to Creating a Data Driven Culture appeared first on Ronald van Loons.


Revolution Analytics

In case you missed it: May 2018 roundup

In case you missed them, here are some articles from May of particular interest to R users. The R Consortium has announced a new round of grants for projects proposed by the R community. A look back...

...
 

June 18, 2018

Ronald van Loon

The Need for Telecom Service Providers to Reinvent Themselves

The world is developing at a rate of knots, and we are entering a phase where there is an enhanced inflow of data. Looking over this data revolution has put extra pressure on telecom operators to perform well, and give the requisite service. To say that these are interesting times for all telecom operators would be a bit of an understatement, because the era that we are in calls for a lot of added feasibility from telecom operators. Telecom operators are in the midst of numerous interesting breakthroughs, and there is now additional responsibility on them to deliver the goods.

On one end, technology is changing at a rapid pace, and keeping up with it requires a lot of efforts, on the other hand, competition is high, and margins have become thin. Besides this, numerous over the top (OTT) players have jumped into the market, and pose as serious contenders for getting the desired market share and revenue. In short, the telecom operators in the market today have to realize the challenges, and alter their modus operandi, because not changing with the times will ultimately leave them in a tight corner. Telecom operators have thus got to reinvent themselves as service providers, and give users the enhanced experience they are looking for.

Challenges Facing Telecom Operators

There are numerous challenges being faced by telecom operators across the globe. Not only do these challenges serve as a hindrance in what telecom operators are doing, but they also limit the growth that these operators want going into the future. Here we look at some of these challenges and how they impede the growth of telecom operators.

The Decline of Traditional Value Added Services

Perhaps one of the biggest concerns troubling telecom operators is the decline of traditional value added services in the market. With added features and enhanced feasibility, users now expect the best of the best, which means that traditional features don’t make the cut anymore. Keeping in mind the requirements of the users, one can now see an increase in the number of Over-The-Top players in the market. These over the top players have taken center stage and provide all users with the feasibility and speed they require.

New age digital services seem to have overpowered the traditional services being provided by telecom operators, and users are inclined towards using them. Some interesting statistics shedding light on this issue are:

  • SMS represents less than 10 percent of all traffic seen globally. Users, or the end customer, has started going for other means of communication, and SMS doesn’t make the cut anymore.
  • Telecom service revenues will be $702 billion by 2021, which will be less than the $751 billion revenue made by over the top services during that year.

Considering the growing use of technology and changing customer experience, it is necessary for telecom operators to reinvent themselves as digital service providers. While operators currently are aware of all the latest innovations being made in communication and technology, they persist with the same tried and tested model for success. The mere fact that they don’t want to get out of their comfort zones can come and haunt them later. This approach is changing now, as telecom operators are realizing the fact that they need to keep up with the developing digital trends and customer preferences.

Problematic Model

The business model for most of these telecom operators is based on a problematic structure that can hinder their progress going into the future. Telecom operators have to realize that to jump into this era they need to step out of the generic pricing models currently being offered to clients. This includes providing flexible charging options and ensuring content monetization.

The absence of flexible charging, coupled with the lack of personalized content, serves as a hindrance for customers opting for a telecom operator. The lack of a proper charging plan, customized according to a customer’s need, may turn customers down and they wouldn’t proceed with the deal. On the contrary, personalized content and charging plans will be received in a better manner, as customers will get a chance to experience better performance and more personalization. Customized content is the need of the hour, and if it is provided, customers won’t have anything to fret over.

Telecom operators now need to achieve steady growth, while simultaneously enabling a single touch point for all operations, settlements, and partners. Moreover, the provision of unified communication policies for customers across different services is also extremely important. Opting for Digital Business Management seems to be the only plausible option worth proceeding forward with, in the face of these challenges.

Why Digital Business Management?

There are numerous reasons as to why telecom operators should opt for digital business management to be a part of the future. A digital business management platform can assist any telecom operator in the following ways.

  • Identifying relevant services: A digital business management (DBM) system will identify the relevant services needed for targeting any appropriate target group. This will help create enhanced customization meeting the needs of every specific target group.
  • Flexible Charging Methods: Flexible charging methods, as we talked about earlier, are the need of the hour, and telecom operators need to move towards them. A DBM system will pave the way for flexible charging through convenient systems.
  • Gathering Customer Insights: Customer insights are relevant in every industry, especially for telecom operators. A DBM system will unearth these relevant insights and help with customer acquisition. This will reduce the cost of acquiring new customers in the long run, and may put an end to high sums being wasted for getting new customers.
  • Protection of Intellectual property in the long run for all parties involved.

How DBM Works

DBM works in an efficient manner to help simplify partnerships and increase the efficiency of different touch points.

Making Sense of Complex Ecosystems

Telecom operators can now get a clear understanding of the global content ecosystem. The overall telecom ecosystem is simplified through the use of an end-to-end service ecosystem. This end-to-end ecosystem includes services procurement, usage reports and analysis, services operations management, and programming and discovery.

Mapping Customer’s Persona

It is extremely necessary to have a customer’s persona before you create customized services for them. A DBM system will help map a customer’s persona, so that you can get to the processes that matter in the long run.

Ensuring Cost Efficiencies

A DBM system will ensure certain cost efficiencies by helping you target the essential channels required for reaching out to customers. This eventually expands your business, and helps you grow the range of your services. Moreover, content monetization through direct carrier billing will decrease costs and increase efficient price charging.

A one size fits all approach is not pertinent any more, which is why telecom operators need to redefine their services and work towards achieving a better overall service. To this end, it is necessary that they get a DBM system that ensures the move towards a revamped business model that achieves customization and has room for growth in the future.

You can get more information on the current technology ecosystem and what is required of telecom operators by downloading the full report here.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post The Need for Telecom Service Providers to Reinvent Themselves appeared first on Ronald van Loons.

Solaimurugan V.

Top list of Artificial Intelligence in Indian Agriculture - research ideas

#AIforAgri body{ margin: 0 ; padding: 0; } .font2{ font-size: 13px; #font-family:arial,sans-serif #font-family:"Comic Sans MS"; } .round{ width:8%; height: 6%; display: inline-block; border-radius: 50%; } .font3{ font-size: 15px; font-family: monospace; #text-shadow: 1px 1px 1px #3D4C4C; } #
 

June 16, 2018


Simplified Analytics

Digital Transformation in Recruitment

A few years ago, the impact of digitization was only established in top industries like Banking, Insurance, and Retail. Now times have changed – the recruitment industry is also adopting digital...

...
 

June 15, 2018


Revolution Analytics

Because it's Friday: Olive Garden Bot

Comedy writer Keaton Patti claims this commercial script for a US Italian restaurant chain was generated by a bot: I forced a bot to watch over 1,000 hours of Olive Garden commercials and then asked...

...

Revolution Analytics

Interpreting machine learning models with the lime package for R

Many types of machine learning classifiers, not least commonly-used techniques like ensemble models and neural networks, are notoriously difficult to interpret. If the model produces a surprising...

...
Ronald van Loon

Digital Meets 5G; Shaping the CxO Agenda

The age of technology is way past its nascent stage and has grown exponentially during the last decade. In my role, travelling to events and meeting with thought leaders I am aware of the developments made across different technological fronts. During these events I have had the opportunity to meet and greet some of the brightest marketing minds. One such person, who is playing an imperative role in ensuring the smooth run of things in both Ericsson and across the digital sphere, is Eva Hedfors.

Eva Hedfors, who is the Head of Marketing and Communications at Ericsson Digital Services, is a leading driver in evolving the perception of Ericsson as a partner to operators transformation from Communication Service Providers to Digital Service providers. I had the opportunity to first meet Eva during Ericsson studio tour in Kista, Sweden. During our first meeting I could tell of her knowledgeable insights and the positive vision she had. Just recently I had the opportunity of interviewing her for a topic she will be presenting in a webinar on the 20th of June. The topic of the webinar – Digital Meets 5G; Shaping the CXO Agenda – is up for interpretation, and she did give me details regarding what she is expecting from the webinar, and how she plans to go about answering some of the questions in this regard.

What Steps should be taken by CxO’s for a Smooth Transition to 5G?

Eva shared her insights on how CxOs could prepare for a smooth transition to Digital Service Providers powered by 5G.”The initial 5G technology deployment will target the massive growth of video traffic in networks, but a leading and hard-to-crack requirement for all CxO’s is also to realize the industry potential and find new business growth through 5G. This involves to both innovate and participate in eco-systems, as well as to optimize the route for marketing such 5G services. CxO’s  can take advantage of 5G to address relatively new segments and industry use cases in mission critical IoT as well as Massive IoT.” Eva explained the business models one creates also needs to be up-to-date and should reflect what’s happening in the market. Since the plan for 5G is rather new, most companies and industries won’t know much about it. Hence, it is necessary that decision makers in Telcos to position their existing capabilities towards different industries and using Network Slicing is one way to do that already on 4G.  To capture the potential in 5G, for many CxO’s means focus to create a revamped strategy for billing and charging systems into a Digital Business Support System (BSS). Moreover, a proper infrastructure needs to be provided to ensure that the end consumer gets to experience the technology in a seamless manner. This would help generate positive insights. 5G is here today, and action needs to start from right now!

How to Avoid the Challenges Involved in Digitization?

The first step to avoiding the challenges involved in digitization is to recognize the efforts most customers have to put in place when engaging with their Telco provider. Once these efforts have been quantified, Telcos can take the necessary action. For the customers, touch points should be made accessible, and there should be no hindrance in communication for B2C, B2B and B2B2C customers. Failure to put the right digital IT infrastructure in place, including analytics and Digital BSS, will limit the business potential of 5G. That is why 5G and Digitalization needs to be planned and executed not as individual technology transformation projects, but as one transformation that aligns towards the same overall business objective in each time frame.  Moreover, the technology teams should be motivated to simplify the core network and make it programmable. Eva mentioned that it was imperative for organizations to start already now and simplify the journey from vEPC to 5G Core for proper implementation and monetization of these revamped services.

Research for 5G Readiness across Different Industries

When asked about the research done to analyze the 5G readiness across different industries, Eva mentioned that Ericsson has done several reports on the potential of 5G across industries.

  1. The 5G business potential study by Ericsson analyzes the business opportunities that come from proper industrial digitalization. The report focuses on the opportunities for organizations present in 10 of the major industries including, Manufacturing, Energy and Utilities, Automotive, Public Safety, Media and Entertainment, Healthcare, Financial Services, Public Transport, Agriculture and Retail. There are detailed use cases for these industries present in the research, which may help stakeholders in these industries to make a decision regarding 5G usage.
  2. Another research based study released by Ericsson in this regard is the guide to capturing 5G-IoT Business Potential. The study answers questions pertaining to the selection of industries and what use cases to address. The insights have been collected from over 200 5G use cases that illustrate how operators can start their IoT business now through the use of 5G.

How Can 5G Technology Improve the Customer Experience Offered to existing Customers by Service Providers?

Enhanced Mobile Broadband is one of the major benefits of 5G technology, according to Eva, and it will help service providers enhance the experience they offer to their customers, who continue to increase consuming video on mobile devises . Better performance, reliability and ultra-high speed are some of the examples of the broadening consumer experience that can be provided through the 5G experience. According to a recent ConsumerLab report conducted by Ericsson, more than 70 percent of all consumers identify performance as one of the major expectations they are looking forward to from 5G.

.

What are the Preparations and Biggest Challenges for 5G Readiness?

Through our industry partnerships we do know, many organizations across many industries have started to analyze how 5G will help drive their digital transformation. 5G business models are being crafted to ensure that the implementation is as smooth as possible. The biggest challenge to capturing the market potential for all actors in the industrial eco-systems, including telecom operators, is the investment in technology and business development. Business development will fall along the lines of organizational adaptation, and Eva believes that a proper infrastructure needs to be provided. It is necessary that 5G be provided the right infrastructure for industry wide implementation. Only organizations that have created the right structure and the model required for 5G implementation are ready for the technology. Without organization-wide infrastructure, 5G would be just like a car running without roads and filling stations.

Integrating 5G Technology across Infrastructure

Like we have talked about above, decision makers need to realize the importance of a proper automated structure that spans across all touch points to ensure that there is no hindrance to 5G services adoption. To that end, organizations also need to realize the importance of an architecture evolution strategy. The evolution strategy should seamlessly integrate 5G across the infrastructure and ensure the full flexibility in the handling of billing, charging and customer interaction.

Both IoT and 5G technologies are shaping the digital transformation and transforming all digital architecture by helping organizations evolve their services and infrastructure. 5G particularly brings a new level of characteristics and performance to the mix, which will play an important role in the digitalization of numerous industries. Telecom operators leveraging the power of 5G technologies can gain from financial benefits as well, as a USD 619 billion revenue opportunity has been predicted for these operators in the future. This revenue opportunity is real and up for grabs by operators, but it does require business model development that elevates telecom operators beyond the connectivity play.

For further insights in this regard, and what CxOs need to do for proper facilitation of Digitalization and 5G technology, you can head over to the webinar being hosted by Eva Hedfors and Irwin van Rijssen, Head of 5G Core Program Ericsson the 20th of June.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post Digital Meets 5G; Shaping the CxO Agenda appeared first on Ronald van Loons.

Ronald van Loon

The Importance of Purpose in Technology & Innovation

Digital transformation is the name of the game for incumbents in almost every industry. Leaders are learning that developing the right technological solutions is only half the battle. Without the right culture and capabilities, an organization will never reap the benefits of those solutions—people will not be willing or able to use tools like IoT and AI, nor take full advantage of end-to-end enterprise platforms. At the heart of building the right culture is the need for a shared sense of purpose among people in the organization.

To explore this idea further, I spoke with Linda A. Hill, a Harvard Business School Professor and the co-author of Collective Genius: The Art and Practice of Leading Innovation. Hill, regarded as a top expert in the field of leadership and innovation, helps leaders understand how to build organizations that can innovate time and again and gain a competitive advantage. At the upcoming Liveworx in Boston on June 19th, Hill, a keynote speaker, will discuss the importance of purpose for organizations that seek to innovate and grow in the future.

What is Innovation?

Hill described that too many companies fall into the trap of defining innovation as synonymous with invention. Many see innovation as something new, often a new technology, ideally one that can be patented. But for Hill, to call something an innovation, it must be both new and useful. It must address some challenge or opportunity aligned with the organization’s purpose and ambition.

How Does Innovation Occur?

Hill has seen companies invest significant time and resources into developing or implementing a new digital platform, only to find out later that few embrace the new technology and take advantage of the associated capabilities. Why? They don’t have the right culture in place.

Innovations are almost always the result of collaborative efforts of people with different expertise and points of view. Therefore, an organization must have a receptive environment – one in which people are willing and able to do the difficult collaborative, discovery-driven work required for innovation—otherwise the new tools go underutilized. As we all know, Hill went on, “Innovations are rarely developed full-blown. They often require iterations, missteps, even failures. Without commitment to a shared purpose, people admit they don’t think it is worth it to take the risk to work in new ways with others—often others who are quite different from themselves—like the new big data analytic experts that have joined their marketing or operations teams.”

Hill believes that leaders play an essential role in creating this receptive environment—a key foundation for innovative work. Hill and her colleagues find rather than defining their role as that of the “visionary”, exceptional leaders of innovation see their primary role as creating a context in which others can co-create the future with them. These leaders do not pretend to be “all-knowing,” but instead encourage employees to come together and use the new technologies to fulfill the organization’s bold ambitions to delight their customers by providing highly differentiated or personalized experiences.

Hill said that people can have difficulty letting go of the myth that innovation is the result of the “lone genius.” They think of a brilliant person who has a sudden “flash of insight.” In fact, Professor Hill’s research shows that innovation is usually a collective process. Organizations that seek to foster routine innovation cannot rely on a few “creative” individuals. Instead, they need to draw out the “slice of genius” in each individual. Innovation is voluntary. No one can be forced to make a contribution or solve a problem. People are willing to do the hard work of innovation when they feel part of a culture engaged in something greater than they could ever achieve alone.

Why Does Purpose Matter for Technology & Innovation?

As Hill described, innovation is a collaborative process—most often among individuals with diverse perspectives and experiences. Getting talented and diverse individuals to work together is not easy, especially because truly creative work is often intellectually and emotionally taxing. But, Hill said, organizations that can innovate routinely have this key ingredient that many organizations do not: Purpose. A compelling purpose is what makes people willing to collaborate, to do the hard work of innovation.

Hill says organizations can define their purpose by answering two questions: 1) Why are we together? 2) Who are we? Organizations that wish to achieve innovation and growth must do more than equip themselves with the required technology infrastructure and solutions. Their leaders must focus on communicating a shared sense of purpose that guides people as they go about their day and make decisions.

When working with organizations, Hill finds that purpose is often misunderstood. Leaders talk of mission statements or strategy. They often focus on what a group does, but not on the why it does what it does. The purpose is more than a goal – it is the reason that the organization exists and the need that it fulfills. Every group, especially in technology, must ask itself: if we disappeared tomorrow, how would the world be different? If your group cannot answer that question, then it might be time to reflect on your purpose and consider how to define it more clearly.

To that end, Professor Hill cited the example of Luca de Meo, the former Chief Marketing Officer for a leading automaker. When De Meo joined the company, innovation was seen as the sole purview of design engineers. This narrow definition of innovation—as something belonging only to the technologists—was slowing the time it took to bring new car designs to market. De Meo knew that the organization needed to be more nimble and that could only happen with breaking down silos—both functional and geographic. The company needed every group engaged in innovative problem-solving together, to include marketing, if they were to be agile and thrive. For sure they needed to build the digital infrastructure and processes to facilitate global collaboration. But without a shared purpose to which all could commit and direct their talents and passions, they would not be willing to create and deliver an innovative brand and unique experience for their customers.

De Meo understood that the first step of their transformation journey was defining a shared purpose—a collective identity that represented the brand. For De Meo and his colleagues, purpose ended up being jointly defined as providing customers with environmentally sustainable mobility options. For a luxury fashion brand, another example cited in Hill’s book, purpose was about delivering products that helped women feel empowered. No matter the form that it takes, purpose is the glue that holds people together in an organization. It is the grease that makes it worth it.

What Values Do Innovative Communities Display?

In addition to purpose, leaders of innovation also instill shared values in their organization. These values shape the organization’s priorities and choices as well as influence individual and collective action. The four values are:

  • Bold ambition. Innovative organizations tackle big challenges (address customer’s key pain points or provide unparalleled experiences that delight them) that conventional ideas could not resolve; in fact, their ambitions tend to be bigger than the company’s capabilities. If the ambition is not bold enough, the purpose important enough, why bother trying to innovate. As one leader of an outsourcing company put it, “If you are going to jump, JUMP!”
  • Collaboration. Innovative organizations take conscious, proactive steps to ensure diverse people interact, now how to debate, and do decision-making in ways that even opposable ideas can be combined and pursued.
  • Learning. Innovative organizations believe a desire to learn enables people to tackle tough problems and benefit from the missteps and mistakes inherent in innovation.
  • Responsibility. Team members feel a sense of responsibility to do their best work to contribute to the organization’s shared purpose.

What’s Next?

The rapid pace of technological change demands constant innovation and growth. Without building the right culture, however, organizations may find that the digital platforms and technological solutions in which they invest a great deal of time and resources may not succeed. Leaders must focus on creating a compelling sense of purpose within their organizations if they wish to achieve routine innovation – whether in technology or other areas. Technical solutions that are implemented in an organization will thrive when the culture supports people in using those tools to pursue innovations that are compelling and meaningful to themselves and their clients.

If you would like more insights into how to foster innovation and growth in your organization, then join Professor Hill and me at the PTC Liveworx.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post The Importance of Purpose in Technology & Innovation appeared first on Ronald van Loons.

 

June 14, 2018


Revolution Analytics

Detecting unconscious bias in models, with R

There's growing awareness that the data we collect, and in particular the variables we include as factors in our predictive models, can lead to unwanted bias in outcomes: from loan applications, to...

...
 

June 13, 2018


Revolution Analytics

Hotfix for Microsoft R Open 3.5.0 on Linux

On Monday, we learned about a serious issue with the installer for Microsoft R Open on Linux-based systems. (Thanks to Norbert Preining for reporting the problem.) The issue was that the installation...

...

Datameer

Regulatory Compliance as a Strategic Weapon – conclusion

Last week we looked at The Ever Evolving Regulatory Environment and the need for a regulatory compliance architecture. If you didn’t see the first blog post on the topic, read it here. Part 4:...

...
 

June 12, 2018

Ronald van Loon

Real-time Data Integration: The X-Factor in Delivering the Ultimate Customer Experience

The age of digital business requires every company to re-think how they rapidly adapt or risk extinction. The presence of unlimited data and the growing needs of customers demand that the customer experience substantially increases. With a wide network of IoT devices and global mobile adoption the world is online all the time, and there is a real-time flow of data coming our way. This drives a new opportunity for businesses to integrate this real-time fast moving data with their core operational data to predict rapidly developing trends and deliver engaging digital experiences providing the ultimate customer experience.

Current Customer Implications in Cloud

There are many challenges to overcome to achieve the promise of real-time data integration. The first of these is the presence of many different data sources. Real-time data is coming in from numerous sources and servers, adding to the complex management structure of the data. Moreover, most businesses are contemplating the move to the cloud, which not only requires the basic infrastructure, but also the integration of more data which is now in multiple cloud silos. With the confusion regarding data sources and management, there is huge pressure on the head of data architects, cloud architects, and DBAs to setup a scalable infrastructure across the board. Even if they are able to create a scalable end infrastructure, its application does not assist them in improving the customer experience.

Enhancing the Customer Experience

There are numerous key factors that are required for the proper management of data within every organization. While we looked at the implications present in real-time data integration above, we’ll now discuss what is needed to improve the customer experience.

  • Access to data/data inventory: Businesses need to unify their core operational data access so it appears as one global data platform. The platform should also assist them in filtering, masking, and transforming data to meet their own expectations.
  • Governance: There needs to be a stringent check over who can see and use the data. Governance of data is an important factor in data management, and special steps are required for ensuring it. Governance has become even more difficult with the addition of data sources from the cloud, as businesses now find it complicated to manage who can access and use what data.
  • Compliance: Complying with government regulations pertaining to customer data is a need now. Recent European Union General Data Protection Regulation (GDPR) regulations make it important for every organization to identify what is being stored, and to keep customers in the loop over how their data is being used and for what purposes.
  • Scalable: Data for real-time integration needs to be scalable from point of contact to deployment. As the compliance and governance fundamentals are in place, ability to dynamically and securely select data to synchronize with a global point of presence or edge computing creates real-time responsiveness and experiences.

Smart Grid Technology

Having talked about what is needed for real time data integration, we now move towards smart grid technology. This smart grid, the first of its kind, can be used for enterprising data coming from your customers. This type of service can be accessed at numerous public cloud servers, and provides data migration and synchronization options that have a positive impact on the customer experience including:

  • Continuous Data Synchronization: This enhances the customer experience by keeping data synchronized within your analytics mechanism. The rapid synchronization of data keeps the system updated at all times. A short explanation how it works. The smart grid synchronizes transactions through the means of the Databus memory transaction replication technology. Synchronization is done at sub-second latency through the use of the Databus.
  • Data Customizations: Data cleansing and customization is an important part of data analysis for every service provider, which is why the data needs to be properly filtered. This provides filtration for users looking to achieve it. You can filter, mask, and encrypt your data according to your own customization needs. All customization services are performed without impacting the data sources.
  • Choice of Topology: You can now customize your data analytics topology according to your own needs. Create your own sophisticated topology with the smart grid technology. Incorporate any one out of the one to many, many to one, and many to many technologies within a matter of minutes.
  • Distributed Architecture: A distributed architecture facilitates the connection of numerous data sources to multiple data destinations. This ensures the smooth flow of data and eradicates any lag time in data collection.

In a time where there are more data points to analyze along the customer journey than ever before, enterprises need to make sure they are not limited by their own data integration capabilities.

Smart grid technology provides a modern data integration platform for enabling the analysis of the right data, at the right time in the right places to impact the customer experience and create new revenue opportunities.

Griddable.io, welcomes you to the world of customization. Their smart grid technology solves the implications we mentioned above. By helping in real-time data integration, Griddable.io helps improve the customer experience for many businesses.

You can register for a free demo (www.Griddable.io/demo) on the Griddable.io smart grid technology and find out the benefits of real-time data integration for yourself.

Ronald

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

More Posts - Website

Follow Me:
TwitterLinkedIn

Author information

Ronald helps data driven companies generating business value with best of breed solutions and a hands-on approach. He has been recognized as one of the top 10 global influencers by DataConomy for predictive analytics, and by Klout for Data Science, Big Data, Business Intelligence and Data Mining and is guest author on leading Big Data sites, is speaker/chairman/panel member on national and international webinars and events and runs a successful series of webinar on Big Data and on Digital Transformation. He has been active in the data (process) management domain for more than 18 years, has founded multiple companies and is now director at a Data Consultancy company, leader in Big Data & data process management solutions. Broad interest in big data, data science, predictive analytics, business intelligence, customer experience and data mining. Feel free to connect on Twitter or LinkedIn to stay up to date on success stories.

The post Real-time Data Integration: The X-Factor in Delivering the Ultimate Customer Experience appeared first on Ronald van Loons.

InData Labs

6 Data Collection Rules for Your Future Perfect Machine Learning Dataset

Modern companies produce gigantic amounts of data. Later it becomes a part of their machine learning datasets. Those are further used to build models that aim to solve various problems business may face, and make it more profitable, customer-oriented and, of course, data-driven. Machine Learning depends heavily on data, that makes algorithm training possible.  Regardless...

Запись 6 Data Collection Rules for Your Future Perfect Machine Learning Dataset впервые появилась InData Labs.

decor