The process of inspecting, cleaning, transforming, and modelling data with the objective of discovering useful information, arriving at conclusions, and supporting the decision making process is called Data Analysis. There are multiple facets and approaches with diverse techniques for the data analysis. The data analysis in statistics are generally divided into descriptive statistics, exploratory data analysis (EDA), and confirmatory data analysis (CDA). Data need to be cleaned. Data cleaning is the process of correcting the outliers and other incorrect and unwanted information. There are several types of data cleaning process to employ depends on the type of data to be cleaned. For quantitative data methods the outlier detection can be used to get rid of anomaly in the data. Spellcheckers can used to lessen the amount of mistyped words in case of textual data.
Business intelligence covers the data analysis that run heavily on aggregation, disaggregation, slicing and dicing, focusing on the business information. Predictive analytics is the application of statistical or structural models for predictive forecasting. Text analytics is the application of statistical, linguistic, and structural models to extract and classify information from texts. All these are varieties of data analysis.
1.Data Cleaning: This is the first process of data analysis where record matching, de-duplication, and column segmentation are done to clean the raw data from different sources.
2.Quality Analysis: Using frequency counts, descriptive statistics such as mean, standard deviation, median, normality histograms such as skewness, kurtosis, frequency, where the n variables are compared with variables of external to the data set.
3.Quality of Measurement: Using confirmatory factor analysis and Analysis of homogeneity.
4.Analysis: There are many analyses which can be done during the initial data analysis phase.
- Univariate statistics ,single variable.
- Bivariate associations correlations.
- Graphical techniques scatter plots.
- Nominal and ordinal variables.
- Frequency counts in numbers and percentages.
- Associations
- Circumambulations
- Hierarchical log linear analysis
- Log linear analysis for the identification of important variables and possible confounders.
- Exact tests or bootstrapping in case of subgroups are small
- Computation of new variables.
- Continuous variables.
- Distribution.
- Statistics – M, SD, variance, skewness and kurtosis.
Main Data Analysis
1.Using exploratory and confirmatory approaches:
In an exploratory analysis no clear hypothesis is stated before analysing the data, and in a confirmatory analysis clear hypotheses about the data are tested.
2.Stability of results:
Stability of results using cross validation,sensitivity analysis and statistical methods.
3.Analysis using different statistical methods
- General linear model: The different statistical models which the general linear model are based are t test, ANOVA, ANCOVA, MANOVA, MANCOVA, ordinary linear regression and F-test. This is multiple linear regression model’s generalisation to the case of more than one dependent variable.
- Generalised linear model: This is an extension and generalization of the general linear model for discrete dependent variables.
- Structural equation modelling: These are sable for assessing latent structures from measured manifest variables.
- Item response theory: Models for used for assessing one latent variable from several binary measured variables.
In the next brief we will take a deep dive into the Data Analysis element of the predictive analytics process.
Quantscience 