In this article we are looking at how can Microsoft Fabric help the aviation industry. After just completing a Proof of Concept (PoC) for an airline so that they can see the power of the new Microsoft technology, Microsoft Fabric. I wanted to expand on this to think about how MS Fabric could help the aviation industry. This includes airlines, airports, and airport services. Datahub Consulting are experts in aviation data and have worked with global airlines and airports for over 5 years with data engineers, data scientists, and data analysts all with a background working in aviation.

In this article we will explain about the PoC that we created along with some other scenarios where Microsoft Fabric could be used. In this article you will see that there are many applications where MS Fabric could be used. As an aviation focused data expert, you will see why I’m excited about this technology. This is not a technical article, so if you don’t have a data background you will be able to follow the steps that we took and how we achieved our goals.

So, what is Microsoft Fabric?

Fabric is the latest technology from Microsoft that was put into general availability last week. When we created the PoC Fabric was in public preview and was a trial version. MS Fabric is a unified platform that brings together cloud storage, called “OneLake”. OneLake is a centralised data lake that is front and centre of the whole platform.

Fabric also integrates.

• Data engineering – prepare and transforming the data using automated processes,
• Data science – machine learning at large scale,
• Data warehousing – for large scale data modeling,
• Real time analytics – ability to stream real time data into Fabric,
• Power BI – for interactive reports and dashboards

All in one application. Making it easier to create end to end solutions that include one or more of the above technologies.

About the PoC that Datahub Created.

We recently worked with an airline to create a PoC to help improve customer experience (CX). Using MS Fabric was the perfect application to showcase this as we could ingest multiple data sources, use a data lake for the data, cleanse and model the data, create a machine learning algorithm, and Power BI for a user-friendly dashboard.

Scope of the PoC

For the scope of the PoC we allowed a 4-week engagement.
Week 1 was to engage with the team at the airline to agree the deliverables, the data for the PoC, the method of delivery, and to setup and import the data into the OneLake.
Week 2 was focused on the data engineering. This was fundamentally to cleanse, prepare and model the data so that all the data from the various sources could be joined into one model.
Week 3 is building ML algorithm and starting the semantic model. There has been a name change for the Power Bi dataset, this is now called the semantic model.
Week 4 was all about finalising the ML model, and review PoC output, building a dashboard with meaningful insights.

The diagram above shows the applications within MS Fabric that we would use for a full end to end project like the PoC in this article. In the PoC we didn’t use the real time analytics due to time constraints but as a project for an airline this could be utilised.

Data Activator is an application to trigger actions based on events. This could  be used to send email notifications or trigger a similar actions.

Data used in the PoC

For the PoC we only included 5 routes for the airline. These were all from the airport where the airline is based (the hub), to 5 destinations with flight times of between  2-5 hours.

Data sources

Internal Data

• Reservation data from Amadeus
• Operational data for flight delays
• Reservation control data
• Frequent flier program (FFP)
• Customer complaints data
• Passenger preference data
• Codeshare data (Due to time constraints codeshare data was not included in this PoC)

External Data

• Weather data (could be included. Due to time constraints weather data was not included in this PoC)

Having the ability to ingest this data into one data lake and having the ability to combine this into one data model allowed the airline to see the data in a new way. Previously this was analysed separately and provided useful information. But allowing the airline to see the relationship between the data in one model gave greater insights into the information. Improving decision making and deeper insights.

There’s the overview, so let’s dig a little deeper into this.

Reservation Data
Most airlines use systems like Amadeus or Sabre as the transactional data for the reservation system. This provides a lot of information on the passengers like where they are traveling from and to. Together with age, nationality, who they are travelling with, and the class of seat, is it economy, business class, or first class. From this information it also can be determined the type of trip. Is it a family vacation or business trip for example.

Operational Data
So, we have the reservation data covered. Now let’s look at operational data. With the goal of improving customer experience then we all know that flight delays will annoy passengers. Will it affect a connecting flight, how long are they going to be waiting at the airport, are they missing quality beach time on the vacation. These are key factors when it comes to customer experience. For this reason, we factored in the flight delay information with data like date and time of the delay, the reason for the delay, which airport the delay happened, how many passengers it affected, and how many passengers had connecting flight.

Reservation Control
Next, reservation control data allowed us to analyse flight changes. Were the flight times changed. This could affect the passenger if they have made prior arrangements for hotel or airport parking. Hotel and parking can be changed by the passenger, but it all adds to the experience score. Also, if there were changes to the flight how much notice was the passengers given, days, weeks, months?

Loyalty Program
The frequent flier program (FFP) is the airlines loyalty program where passengers can become a member to get benefits like air miles. We used this data to understand the number of passengers that were frequent fliers. If a passenger is a regular flier with the airline, then it’s reasonable to assume that the customer experience would be better. Nobody would join as a frequent flier with an airline where they had a bad experience.

Customer Complaints
With customer complaints this was key to the improving customer experience. If the passenger felt that they needed to raise a complaint, then this would affect the customer experience. The data that we analysed was the number of complaints, flight concerned, reason for the complaint, the resolution, time to resolve, and if the passenger received any compensation.

Passenger Preferences
Then finally the passenger preferences. Does the passenger have any preferences? This could be that they may have a food allergy, do they prefer a window seat, does the passenger use the VIP lounge, do they book any additional services like airport transfers etc.

Other Data

Let’s talk about the data that could be also includes to make this solution even better. As this was a PoC and we had enough data. We did not have to include weather or codeshare data. Codeshare is where an airline partners with another airline who can fly all or part of a route. Let me explain. In a scenario, if we had an airline in Germany that only fly to European destinations. If that airline wants to sell transatlantic flights to U.S. to sell more routes, then this would only be possible if they partnered with another airline that can do part of the route. it would all be sold under one reservation but the passenger would get a connecting flight.

Data engineering

We used MS Fabric to look at the quality of the data, select the data for the 5 routes that we wanted to use. Cleanse, prepare, and join the data together with either passport number, flight number, FFP number. Then we were able to see the data in a better way.

Machine learning

To assess the customer experience we used sentiment analysis. This is where a algorithm would cluster customers based on a prediction. The passengers would have a sentiment score that would then put them in a category ranging from a very positive experience through to a very negative experience based on the algorithm.

The sentiment analysis scoring was set up to split the customers into 5 categories. The airline could look further into anyone that was considered neutral or lower, starting with the very negative.

• Very positive
• Positive
• Neutral
• Negative
• Very negative

Please be mindful that with sentiment analysis, it can be subjective based on demographics, cultural, and regional differences. What could be acceptable in one country may not be in another. For that reason with a project like this, the machine learning is not a one model fits all.

Power BI Dashboard

Power BI is a dashboard and reporting application that can transform data into interactive insights.

Dashboards and reports, any time, any where, on any device.

Create rich, interactive data visualisations from multiple data sources and share important business insights that drive success with Power BI

Bridging the gap between data and decision making.

PoC Outcome

At the time of writing this article the airlines was evaluating the model and assessing a plan to potentially roll this PoC out to all routes and passengers. Being given the analysis of customer experience, this then allows the airline to make decisions that will enhance the customers experience.

Once in production, with all customers included the machine learning will mature and better predictions will be available. From the sample data included we were getting a 94.2% accuracy. Based on the development time and data included this was a good success rate.

Why is MS Fabric different?

There are many advantages to MS fabric both from a technical, and also delivery / project management perspective. For example,

• Where previously data engineers, data scientists, data analysts all working on one project, but each working in their own applications in separate workspaces. Now this isn’t the case, and the collective development team can all work more collaboratively on a single platform. From experience this would make delivering a project easier and quicker.
• Having one unified platform also supports end to end data governance, data protection, and IT security (ISO 27001).

Other aviation use cases

As well as improving passenger experience, there are many other use cases where MS Fabric could be used.

Fare Optimisation
Have you looked at a flight and seen the price go up and down.
Pricing strategy is a complex subject and to explain it in detail would require an article by itself. But availability and demand are two factors that are used within the pricing of seats. Allowing a machine learning algorithm to set the price of tickets is key to optimising ticket pricing strategy.  

Reservation and revenue analysis
It’s been reported by CNBC news that the impact of Israel’s war against Hamas in the Gaza Strip has had a negative impact on airlines. It’s reported that ticket sales and revenue of many airlines will be affected. MS Fabric could be used to look at routes that an airline has available, geographic location of the airline etc to understand the impact that external factors can play.
Dubai airshow: Airlines are feeling the impact of the Israel-Hamas war (cnbc.com)

Using a different scenario, weather data could also be analysed in the same way. With climate change weather is becoming more unpredictable, with trend analysis and machine leaning, weather data predictions are more accurate. Again this will have an impact on ticket sales and revenue.

Realtime analysis
It’s possible to stream real time data into MS fabric. These are many applications where this could be used.

• Let’s look at airports first, Using CCTV and surveillance cameras, in real time the airport could be monitored to look at volumes of passengers and where there could be bottlenecks. For example, what are the volumes of passengers going though security, or at the check-in desks.
• It could be global data steamed from Google Analytics where the airline could collect information on website traffic. This could be monitored in real time to understand trends in website activity based of geographic location.
• It also could be used with aircraft ADS-B data that can monitor the aircraft in real time. Also, if the aircraft transponder squawks an emergency code this could trigger an alert via MS Fabric. In this instance information like latitude, longitude, altitude, and heading could be included in the alert.
  
  Transponder emergency codes are:
  7500 – Aircraft Hijacking
  7600 – Radio Failure
  7700 – Emergency

Planned maintenance
An airline is only making money when an aircraft is in the air. The more time that the aircraft is on the ground, the airline is potentially losing revenue. In some cases, it will cost them money.
Predictive maintenance is looking at the individual parts on an aircraft and proactively carrying out maintenance so that failures on an aircraft are kept to a minimum. When a part or system on an aircraft fails it’s then grounded until the problem is fixed. This can cause delays, cancellations, airlines may have to pay additional fees to passengers and airports, and customer satisfaction is then reduced. I’ve personally travelled with a particular airline 7 times. Out of those 7 occasions the plane was delayed 5 times. 3 of those times were due to a problem with the aircraft.
Machine learning can look at an individual aircraft and use the data to predict the potential failure point of a part or component. Knowing this information, the airline can then plan for the part to be inspected or replaced. With planned maintenance this does not cause the airline any disruption and can minimize the cost impact.

Safety & risk management
At airports it’s important to monitor safety and risks.

• Health & safety incident reporting.
  With the high volume of people using airports daily there will be regular health and safety incidents. This can be analysed and managed efficiently. For example, how many incidents required a paramedic to attend, and how long did it take to respond. How many incidents caused flight delays and how can this be reduced.
• Analysis of root cause of incidents
  With any incident, especially at airports, the management team would want to know root causes and ways to reduce the incidents. This again can be achieved with analytics.

Flight operations
Predicting Delays
With flight operations MS Fabric can help with predicting delays. There are various data that can used to help detect potential delays and their impact. This includes both internal airline data, but also external data. Weather data would be key to predicting delays. For example, on the 19^th November 2023 it was reported by the National Air Traffic Service (NATS) that high winds led to temporary flight restrictions, that in turn caused delays at London Heathrow airport (LHR). With weather either high winds or storms could impact delays. Using MS Fabric combining weather data with other data points would allow for machine learning algorithms and alerts for airports and airlines in advance of any predicted delays.

Optimising Workflows
As airlines and airports are 24/7 operations optimising any workflow can improve efficiencies. This could be alerting staff to delays. Another example would be with the CCTV monitoring of vehicles or people. With people, CCTV images could be used with digital transformation tools to understand and optimise footfall in particular parts of an airport.

Using CCTV images cars dropping off passengers at the airport could be analysed and alerts sent to staff or departments based on congestion.

In Summary

Hopefully this article provides a great insight into how Microsoft Fabric can support airports and airlines improve their business. Coupled with the expertise and aviation knowledge of Datahub Consulting, this provides a winning team.

Working with Datahub Consulting

Datahub are experts in delivering data solutions within the aviation industry. Regardless of if you have data teams within your business, we can add value and success. Either delivering end to end solutions or supporting your internal team to deliver success.

It wouldn’t cost you anything to start a conversation with our CEO that has personally worked with airlines like South African Airways, Middle East Airlines, Oman Air, Flynas, and Saudia.

Contact us
Contact us | DataHub Consulting
Datahub Consulting Website
Data Consultancy Services | Datahub Consulting
Datahub Aviation Center of Excellence
Aviation Center of Excellence | DataHub Consulting