SSIS – CONFIGURE ENVIRONMENTS AND ENVIRONMENT VARIABLES

As we see above, we can create folder in SSISDB. Each folder we create, it will automatically create two subfolders called Projects andEnvironments 
Projects: A place holder for deployed projects inside Folder
Environments: A placed holder for environments like DEV and QA
To create an environment, right click on the Environments subfolder and click on Create Environment menu item as shown below:




















SSIS – DEPLOY DEVELOPED SSIS PROJECT TO INTEGRATION SERVICES CATALOG

After development of our SSIS package to SSISDB, to deploy SSIS Project,just right click on project file and choose Deploy

















SSIS – DEVELOP SIMPLE SSIS PROJECT

SSIS Project creation remains same from SSIS 2008 to SSIS 2012. To create SSIS project file, we need to open BIDS and choose Integration services .If you are new to SSIS , please go through basis of SSIS project creation and package development
In my example,  I have two similar tables (CountryDetails) in two different databases (SampleDataBase1, SampleDataBase2).  I created one simple package, logic is to insert one row to CountryDetails table in either of SampleDataBase1, SampleDataBase2 databases based on environment configuration

Table Structure:

CREATE TABLE [dbo].[CountryDetails](
       [ID] [float] NULL,
       [StateProvinceCode] [nvarchar](255) NULL,
       [CountryRegionCode] [nvarchar](255) NULL,
       [IsOnlyStateProvinceFlag] [float] NULL,
       [Name] [nvarchar](255) NULL,
       [TerritoryID] [float] NULL

) ON [PRIMARY]

SSIS – DEPLOYED SSIS PROJECTS VERSIONS AND RESTORE TO PREVIOUS VERSION

Good thing in Project Deployment Model is we can store multiple versions of SSIS projects inside SSISDB. And also if we want to rollback to previous versions of deployed packages, we can simply restore to desired version. To see the previous versions of deployed projects, just right click on project and choose Versions 

























SSIS – REDEPLOYING THE PROJECT TO INTEGRATION SERVICES CATALOG

Nothing is constant ,we usually gets so many change requests in our process with respect to Functionality, Performance and  fix fixes .So we need to change our Packages according to new requirements.  Once we changed our SSIS packages in Project and if we want to redeploy SSIS packages to same location in Integration Services Catalog, just right click on Project and choose Deploy option. Redeploying Packages is not different from deploying SSIS packages first Time. 





















SSIS – VALIDATE THE DEPLOYED PROJECT OR PACKAGE

Once we deploy the SSIS Packages to Integration Services Catalog, we need to validate before we execute our packages to ensure that they will execute properly. In validation, it will show list of errors if we have any issues with our packages


SSIS – HOW TO OPEN .ISPAC IN BIDS (VISUAL STUDIO)

Sometimes we need to edit the deployed SSIS packages, so to edit we need to open deployed .ispac in BIDS. We can open .ispac in two ways by using integration services import project wizard

SSIS – EXECUTE DEPLOYED PROJECT/PACKAGE WITH THE ENVIRONMENT

Once we develop and deployed SSIS project to Integration Services Catalog, configured environment/environments variables and given reference for environments. We need to unit test by executing deployed packages, whether our environments variables inside environments are consuming or not. 

SSIS – CONFIGURE ENVIRONMENT REFERENCE IN THE DEPLOYED PROJECT

We have configured Integration Services Catalog and deployed project and created two environments. Now we need to make relation between environments and deployed projects. To do that, right click on deployed project and choose configure 
















SSIS – PROJECT DEPLOYMENT MODEL

As a MSBI developer, I always enjoys while deploying of SSRS reports and SSAS cubes but not in case of SSIS packages deployment because we have project level single click deployment in SSRS and SSAS, here our life is so easy but I always feel difficult with  SSIS packages legacy deployment and management of SSIS packages. There we need to deploy SSIS packages individually either in file system or system database MSDB
Good News for SSIS Developers is, in SQL server 2012 Microsoft introduced SSIS Package Deployment Model i.e. single click project deployment. Also it introduced new specific database for only SSIS Package Deployment called SSISDB. Now in this topic I cover what it is SSIS Package Deployment Model and how to deploy your SSIS packages to SSISDB

SSIS – INTERVIEW QUESTIONS

1.  Package Configurations: IF SQL Server Package Configuration, How do implement it    and what are the things need to change when moved to other    Environments.
2.  Get file names from the folder using SSIS. (Can use ForEach loop Container and Change the propery to only "file Name" or "File and Extension")

SQL SERVER – HOW TO UPDATE SQL JOB STEP SCRIPT DYNAMICALLY

Generally we create SQL Server Jobs to execute some SQL scripts, SSIS packages or power shell commands when user calls job manually or call automatically by Scheduler. Sometimes we need to change SQL server Job steps script dynamically based on some dynamic values. Now let’s see, how to change SQL server Job steps command dynamically
To show this, I have taken example of SQL Job step with T-SQL Command. Before this I have created one simple stored procedure in ADVENTUREWORKSDW2012 DB to insert values into DIMPRODUCT table.

SQL SERVER – HOW TO DELETE ALL TABLES DATA

By using below script we can able to delete all tables data in SQL SERVER without having any issues. In the script I have followed below steps

·         Disable all constraints on all tables. Due to that we can delete all tables without having any constraints issues by using system stored procedure sp_MSForEachTable
·         Disable all TRIGGERs on all tables
·         Delete all tables data
·         If any tables having identity, we need to set identity to 0
·         Once we delete all tables data , we will enable all CONSTRAINTS
·         Once we delete all tables data , we will enable all TRIGGERs

SQL SERVER – SEARCH RECORDS WHICH ARE SEPARATED WITH PIPE DELIMITER


Today I got phone call from one of my friend regarding some query to fetch records from one table let's called Person table and column Role  .But Column Role having multiple values separated with comma delimiter . Now he need to find records where input value matches any value in column Role from Person table. Then I have given one stupid answer with long query like by split the record into multiple records based on delimiter and then search record based on input and then club the record. Finally his requirement meets and he is happy but I am very upset with that big query .Then I try to write get same result with some different logic like below.

SQL SERVER – DATABASE MAIL ENABLE AND CONFIGURING USING T-SQL

Today I got requirement like after my ETL packages execution I need to send one email to team regarding what is status of package execution and how many records got success and fail . To do that I decided to use database mail component in SQL Server. We can configure database mail by two ways
1. Using GUI
2. Using SQL Script 
In Previous post Configure DB Mail using GUI, we have seen how to configure DB Mail using GUI. Now we see how to configure using T-SQL

SQL SERVER – DATABASE MAIL ENABLE AND CONFIGURING USING GUI

Today I got requirement like after my ETL packages execution I need to send one email to team regarding what is status of package execution and how many records got success and fail . To do that I decided to use database mail component in SQL Server. We can configure database mail by two ways
1. Using GUI
2. Using SQL Script

SQL SERVER – HOW TO DISABLE AND REBUILD ALL INDEXES IN TABLE


Some time while loading data to tables due to having of indexes, data population makes slow. So while loading data to table we need to disable non clustered indexes and after loading of data we need to rebuild indexes.

Below script will disable existing indexes and rebuild with the parameter of table name

SQL SERVER – DATA PROFILING WITH SQL SCRIPT

CREATE DATABASE [Data_Profiling]
GO

USE [Data_Profiling]
GO

CREATE TABLE [dbo].[Data_Profile_ColumnStatistics] (
[Database_Name] [varchar](100) NULL
,[Table_Name] [varchar](100) NULL
,[Column_Name] [varchar](100) NULL
,[Row_Count] [int] NULL
,[Min_Value] [varchar](50) NULL
,[Max_Value] [varchar](50) NULL
,[Mean] [varchar](50) NULL
,[StdDev] [varchar](50) NULL
,[JobId] [int] NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[Data_Profile_LengthDistribution] (
[Database_Name] [varchar](100) NULL
,[Table_Name] [varchar](100) NULL
,[Column_Name] [varchar](100) NULL
,[Row_Count] [int] NULL
,[Min_Length] [int] NULL
,[Max_Length] [int] NULL
,[Length] [int] NULL
,[Count] [int] NULL
,[JobId] [int] NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[Data_Profile_NULL_Counts] (
[Database_Name] [varchar](100) NULL
,[Table_Name] [varchar](100) NULL
,[Column_Name] [varchar](100) NULL
,[SqlDbType] [varchar](100) NULL
,[Max_Length] [int] NULL
,[Precision] [int] NULL
,[Scale] [int] NULL
,[IsNullable] [varchar](100) NULL
,[Row_Count] [int] NULL
,[Null_Count] [int] NULL
,[JobId] [int] NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[Data_Profile_ValueDistribution] (
[Database_Name] [varchar](100) NULL
,[Table_Name] [varchar](100) NULL
,[Column_Name] [varchar](100) NULL
,[Row_Count] [int] NULL
,[NumberOfDistinctValues] [int] NULL
,[Value] [varchar](max) NULL
,[Count] [int] NULL
,[JobId] [int] NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

--exec Data_Profiling 'DBName'
CREATE PROCEDURE [dbo].[Data_Profiling] (@dbname VARCHAR(50))
AS
BEGIN
--declare @dbname varchar(50)='xyz'
--declare @dbname varchar(50)='DBName
DECLARE @dataprofiling TABLE (
[ColumnName] [nvarchar](128) NULL
,[SchemaName] [nvarchar](128) NULL
,[TableName] [nvarchar](128) NULL
,[ObjectType] [nvarchar](60) NULL
,[DataType] [nvarchar](128) NULL
,[max_length] [smallint] NULL
,[is_nullable] [bit] NULL
,[column_id] [int] NULL
,[column_id_1] [int] NULL
,[object_id] [int] NULL
,[definition] [ntext] NULL
,precision INT NULL
,scale INT NULL
,seed_value INT
,increment_value INT
,last_value INT
,scdefination NVARCHAR(max)
,ccdefination NVARCHAR(max)
)
DECLARE @sql NVARCHAR(max);

SET @sql = 'select
  rtrim(ltrim(s.name)) as ColumnName,
    sh.name as SchemaName,
    o.name as TableName,
    o.type_desc AS ObjectType,
    t.name as DataType,
    s.max_length,
    s.is_nullable,
    ic.column_id,
    sc.column_id as column_id_1,
    cc.object_id,
    cc.definition,
    cast(S.precision as int) as precision,
    cast(s.scale as int) as scale,
   cast(ic.seed_value as int) as seed_value,
   cast(increment_value as int) as increment_value,
  cast(last_value as int) as last_value,
  sc.definition as scdefination,
 cc.definition as ccdefination
     from ' + @dbname + '.sys.tables  tx  join
    ' + @dbname + '.sys.columns                           s on tx.object_id=s.object_id
        INNER JOIN ' + @dbname + '.sys.types                   t ON s.system_type_id=t.user_type_id and t.is_user_defined=0
        INNER JOIN ' + @dbname + '.sys.objects                 o ON s.object_id=o.object_id
        INNER JOIN ' + @dbname + 
'.sys.schemas                sh on o.schema_id=sh.schema_id
        LEFT OUTER JOIN ' + @dbname + '.sys.identity_columns  ic ON s.object_id=ic.object_id AND s.column_id=ic.column_id
        LEFT OUTER JOIN ' + @dbname + '.sys.computed_columns  sc ON s.object_id=sc.object_id AND s.column_id=sc.column_id
        LEFT OUTER JOIN ' + @dbname + '.sys.check_constraints cc ON s.object_id=cc.parent_object_id AND s.column_id=cc.parent_column_id where s.name not like ''%-%'' and s.name not like ''%[%'' and s.name not like '' ''';

INSERT INTO @dataprofiling
EXEC (@sql)

DECLARE @BIP_sourcetable TABLE (
[ColumnName] [nvarchar](128) NULL
,[ObjectName] [nvarchar](257) NULL
,[ObjectType] [nvarchar](60) NULL
,[DataType] [nvarchar](151) NULL
,[Nullable] [varchar](8) NULL
,[MiscInfo] [ntext] NULL
,[DatabaseName] [nvarchar](100) NULL
,length INT
,precision INT NULL
,scale INT NULL
,[is_nullable] [bit] NULL
)

INSERT INTO @BIP_sourcetable (
[ColumnName]
,[ObjectName]
,[ObjectType]
,[DataType]
,[Nullable]
,[MiscInfo]
,DatabaseName
,length
,precision
,scale
,[is_nullable]
)
SELECT '[' + RTRIM(LTRIM(ColumnName)) + ']'
,SchemaName + '.' + '[' + TableName + ']' AS ObjectName
,ObjectType
,CASE 
WHEN [DataType] IN ('char', 'varchar')
THEN [DataType] + '(' + CASE 
WHEN max_length < 0
THEN 'MAX'
ELSE CONVERT(VARCHAR(10), max_length)
END + ')'
WHEN [DataType] IN ('nvarchar', 'nchar')
THEN [DataType] + '(' + CASE 
WHEN max_length < 0
THEN 'MAX'
ELSE CONVERT(VARCHAR(10), max_length / 2)
END + ')'
WHEN [DataType] IN ('numeric')
THEN [DataType] + '(' + CONVERT(VARCHAR(10), precision) + ',' + CONVERT(VARCHAR(10), scale) + ')'
ELSE [DataType]
END AS DataType
,CASE 
WHEN is_nullable = 1
THEN 'NULL'
ELSE 'NOT NULL'
END AS Nullable
,CASE 
WHEN column_id IS NULL
THEN ''
ELSE ' identity(' + ISNULL(CONVERT(VARCHAR(10), seed_value), '') + ',' + ISNULL(CONVERT(VARCHAR(10), increment_value), '') + ')=' + ISNULL(CONVERT(VARCHAR(10), last_value), 'null')
END + CASE 
WHEN column_id_1 IS NULL
THEN ''
ELSE ' computed(' + ISNULL(scdefination, '') + ')'
END + CASE 
WHEN object_id IS NULL
THEN ''
ELSE ' check(' + ISNULL(ccdefination, '') + ')'
END AS MiscInfo
,@dbname
,max_length
,precision
,scale
,[is_nullable]
FROM @dataprofiling

DECLARE @temptable TABLE (
id INT IDENTITY(1, 1)
,[ColumnName] NVARCHAR(100)
,[ObjectName] NVARCHAR(100)
,DatabaseName NVARCHAR(100)
,DataType VARCHAR(100)
,length INT
,precision INT NULL
,scale INT NULL
,[is_nullable] BIT
)
DECLARE @temptable1 TABLE (
id INT IDENTITY(1, 1)
,[ColumnName] NVARCHAR(100)
,[ObjectName] NVARCHAR(100)
,DatabaseName NVARCHAR(100)
,DataType VARCHAR(50)
,length INT
,precision INT NULL
,scale INT NULL
,[is_nullable] BIT
)
DECLARE @ColumnObjectName TABLE (
id INT IDENTITY(1, 1)
,ColumnName NVARCHAR(100) NULL
,ObjectName NVARCHAR(100) NULL
,DataType VARCHAR(50)
,DBName VARCHAR(50)
,length INT
,precision INT NULL
,scale INT NULL
,[is_nullable] BIT
)
DECLARE @ColumnObjectName1 TABLE (
id INT IDENTITY(1, 1)
,[ColumnName] NVARCHAR(100) NULL
,[ObjectName] NVARCHAR(100) NULL
,DataType VARCHAR(50)
,DBName VARCHAR(50)
,length INT
,precision INT NULL
,scale INT NULL
,[is_nullable] BIT
)
DECLARE @minMax TABLE (
id INT IDENTITY(1, 1)
,Minvalue FLOAT
,MaxValue FLOAT
,AvgValue FLOAT
,STDEVValue FLOAT
,DistinctCount INT
,Count INT
,ColumnName NVARCHAR(100) NULL
,ObjectName NVARCHAR(100) NULL
,P FLOAT NULL
,NullCount INT
,minLen NVARCHAR(100) NULL
,Maxlen NVARCHAR(100) NULL
)
DECLARE @Count_Distinct TABLE (
id INT IDENTITY(1, 1)
,Minvalue FLOAT
,MaxValue FLOAT
,AvgValue FLOAT
,STDEVValue FLOAT
,DistinctCount INT
,Count INT
,ColumnName NVARCHAR(100) NULL
,ObjectName NVARCHAR(100) NULL
,P FLOAT NULL
,NullCount INT
,minLen NVARCHAR(100) NULL
,Maxlen NVARCHAR(100) NULL
)

INSERT INTO @temptable (
[ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
,precision
,scale
,[is_nullable]
)
SELECT [ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
,precision
,scale
,[is_nullable]
FROM @BIP_sourcetable
WHERE DataType IN ('int', 'float', 'decimal', 'tinyint', 'smallint')

INSERT INTO @temptable1 (
[ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
,precision
,scale
,[is_nullable]
)
SELECT [ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
,precision
,scale
,[is_nullable]
FROM @BIP_sourcetable
WHERE DataType NOT IN ('int', 'float', 'decimal', 'tinyint', 'smallint', 'xml', 'Image', 'ntext')

DECLARE @dbname1 VARCHAR(50) = @dbname
DECLARE @max INT = (
SELECT MAX(id)
FROM @temptable
)
DECLARE @i INT = 1

WHILE @i <= @max
BEGIN
DECLARE @ColumnName VARCHAR(50) = (
SELECT [ColumnName]
FROM @temptable
WHERE id = @i
)
DECLARE @ObjectName VARCHAR(50) = (
SELECT [ObjectName]
FROM @temptable
WHERE id = @i
)
DECLARE @DataType VARCHAR(50) = (
SELECT DataType
FROM @temptable
WHERE id = @i
)
DECLARE @length INT = (
SELECT length
FROM @temptable
WHERE id = @i
)
DECLARE @precision INT = (
SELECT precision
FROM @temptable
WHERE id = @i
)
DECLARE @scale INT = (
SELECT scale
FROM @temptable
WHERE id = @i
)
DECLARE @is_nullable BIT = (
SELECT [is_nullable]
FROM @temptable
WHERE id = @i
)
DECLARE @sql1 NVARCHAR(max)

SET @sql1 = (
'select  min(' + @ColumnName + '),max(' + @ColumnName + '),avg(cast(' + @ColumnName + ' as bigint)) ,STDEV(' + @ColumnName + '),Count(distinct ' + @ColumnName + ' ) as DistinctCount ,
  Count(*) as Count,''' + @columnName + ''',''' + @ObjectName + ''',(100.0 * SUM(CASE WHEN ' + @ColumnName + ' IS NULL THEN 1 ELSE 0 END) / COUNT(*)) as p,
  SUM(CASE WHEN ' + @ColumnName + ' IS NULL THEN 1 ELSE 0 END) as NullCount,Min(len(' + @ColumnName + ')),Max(len(' + @ColumnName + '))  from ' + @dbname1 + '.' + @ObjectName + ''
)

INSERT INTO @minMax (
minvalue
,maxvalue
,AvgValue
,STDEVValue
,DistinctCount
,Count
,ColumnName
,ObjectName
,P
,NullCount
,minLen
,Maxlen
)
EXEC (@sql1)

INSERT INTO @ColumnObjectName (
ColumnName
,ObjectName
,DataType
,DBName
,length
,precision
,scale
,[is_nullable]
)
SELECT @ColumnName
,@ObjectName
,@DataType
,@dbname1
,@length
,@precision
,@scale
,@is_nullable

SET @i = @i + 1
END

DECLARE @max1 INT = (
SELECT MAX(id)
FROM @temptable1
)
DECLARE @i1 INT = 1

WHILE @i1 <= @max1
BEGIN
DECLARE @ColumnName1 VARCHAR(50) = (
SELECT [ColumnName]
FROM @temptable1
WHERE id = @i1
)
DECLARE @ObjectName1 VARCHAR(50) = (
SELECT [ObjectName]
FROM @temptable1
WHERE id = @i1
)
DECLARE @DataType1 VARCHAR(50) = (
SELECT DataType
FROM @temptable1
WHERE id = @i1
)
DECLARE @length1 INT = (
SELECT length
FROM @temptable1
WHERE id = @i1
)
DECLARE @precision1 INT = (
SELECT precision
FROM @temptable1
WHERE id = @i1
)
DECLARE @scale1 INT = (
SELECT scale
FROM @temptable1
WHERE id = @i1
)
DECLARE @is_nullable1 BIT = (
SELECT [is_nullable]
FROM @temptable1
WHERE id = @i1
)
DECLARE @sql2 NVARCHAR(max)

SET @sql2 = (
'select  Null,Null,Null,Null,Count(distinct ' + @ColumnName1 + ' ) as DistinctCount ,Count(*) as Count, ''' + @columnName1 + ''',
  ''' + @ObjectName1 + ''' ,(100.0 * SUM(CASE WHEN ' + @ColumnName1 + ' IS NULL THEN 1 ELSE 0 END) / COUNT(*)) as p,SUM(CASE WHEN ' + @ColumnName1 + ' IS NULL THEN 1 ELSE 0 END) as NullCount,
  Min(len(' + @ColumnName1 + ')),Max(len(' + @ColumnName1 + '))  from ' + @dbname1 + '.' + @ObjectName1 + ''
)

INSERT INTO @Count_Distinct (
minvalue
,maxvalue
,AvgValue
,STDEVValue
,DistinctCount
,Count
,ColumnName
,ObjectName
,p
,NullCount
,minLen
,Maxlen
)
EXEC (@sql2)

INSERT INTO @ColumnObjectName1 (
ColumnName
,ObjectName
,DataType
,DBName
,length
,precision
,scale
,[is_nullable]
)
SELECT @ColumnName1
,@ObjectName1
,@DataType1
,@dbname1
,@length1
,@precision1
,@scale1
,@is_nullable1

SET @i1 = @i1 + 1
END

DELETE
FROM [Data_Profiling].[dbo].[Data_Profile_NULL_Counts]
WHERE [Database_Name] = @dbname

INSERT INTO [Data_Profiling].[dbo].[Data_Profile_NULL_Counts] (
[Database_Name]
,[Table_Name]
,[Column_Name]
,[SqlDbType]
,[Max_Length]
,[Precision]
,[Scale]
,[IsNullable]
,[Row_Count]
,[Null_Count]
)
SELECT DISTINCT t.DataBaseName
,t.TableName
,t.ColumnName
,t.DataType
,t.length
,t.precision
,t.scale
,CASE 
WHEN t.is_nullable = 0
THEN 'No'
ELSE 'Yes'
END AS Nullable
,t.Count
,CASE 
WHEN t.NullCount IS NULL
THEN 0
ELSE t.NullCount
END AS NullCount
FROM (
SELECT c.ColumnName
,c.ObjectName AS TableName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @minMax m
JOIN @ColumnObjectName c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName

UNION

SELECT c.ColumnName
,c.ObjectName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @Count_Distinct m
JOIN @ColumnObjectName1 c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName
) t

DELETE
FROM [Data_Profiling].[dbo].[Data_Profile_ColumnStatistics]
WHERE [Database_Name] = @dbname

INSERT INTO [Data_Profiling].[dbo].[Data_Profile_ColumnStatistics] (
[Database_Name]
,[Table_Name]
,[Column_Name]
,[Row_Count]
,[Min_Value]
,[Max_Value]
,[Mean]
,[StdDev]
)
SELECT DISTINCT t.DataBaseName
,t.TableName
,t.ColumnName
,t.Count
,t.Minvalue
,t.MaxValue
,t.AvgValue
,t.StdValue
FROM (
SELECT c.ColumnName
,c.ObjectName AS TableName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @minMax m
JOIN @ColumnObjectName c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName

UNION

SELECT c.ColumnName
,c.ObjectName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @Count_Distinct m
JOIN @ColumnObjectName1 c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName
) t

DELETE
FROM [Data_Profiling].[dbo].[Data_Profile_LengthDistribution]
WHERE [Database_Name] = @dbname

INSERT INTO [Data_Profiling].[dbo].[Data_Profile_LengthDistribution] (
[Database_Name]
,[Table_Name]
,[Column_Name]
,[Row_Count]
,[Min_Length]
,[Max_Length]
,[Length]
,[Count]
)
SELECT DISTINCT t.DataBaseName
,t.TableName
,t.ColumnName
,t.Count
,t.MinLen
,t.MaxLen
,t.length
,t.DistinctCount
FROM (
SELECT c.ColumnName
,c.ObjectName AS TableName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @minMax m
JOIN @ColumnObjectName c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName

UNION

SELECT c.ColumnName
,c.ObjectName
,c.DBName AS DataBaseName
,DataType
,m.Minvalue
,m.MaxValue
,AvgValue
,m.STDEVValue AS StdValue
,DistinctCount
,Count
,CASE 
WHEN P IS NULL
THEN 100
ELSE P
END AS NullPercentage
,ISNULL(minLen, 0) AS MinLen
,ISNULL(Maxlen, 0) AS MaxLen
,c.length
,m.NullCount
,c.precision
,c.scale
,CAST(c.is_nullable AS VARCHAR) AS is_nullable
FROM @Count_Distinct m
JOIN @ColumnObjectName1 c ON c.ColumnName = m.ColumnName
AND m.ObjectName = c.ObjectName
) t

/*
  -------------------------------------------------------------
  insert into [Data_Profiling].[dbo].[Data_Profile_CandiateKey2]
  (
        [Database_Name]
        ,[Table_Name]
        ,[Column_Name]
        ,[Keycolumn]
   )
  
  
   select distinct t.DataBaseName,t.TableName,t.ColumnName,(t.DistinctCount/t.Count)*100
   from (
  
  
  select c.ColumnName,c.ObjectName as TableName,c.DBName as DataBaseName,DataType,m.Minvalue,m.MaxValue,AvgValue,
  m.STDEVValue as StdValue,DistinctCount,Count,case when P IS null then 100 else P end as NullPercentage,isnull(minLen,0) as MinLen,isnull(Maxlen,0) as MaxLen,
  c.length,m.NullCount ,c.precision,c.scale,cast(c.is_nullable as varchar) as is_nullable
  
   from @minMax m
  join @ColumnObjectName c on c.ColumnName=m.ColumnName and m.ObjectName=c.ObjectName
  union
  select c.ColumnName,c.ObjectName,c.DBName as DataBaseName,DataType,m.Minvalue,m.MaxValue,AvgValue,
  m.STDEVValue as StdValue,DistinctCount,Count,case when P IS null then 100 else P end as NullPercentage,isnull(minLen,0) as MinLen,isnull(Maxlen,0) as MaxLen,
  c.length,m.NullCount ,c.precision,c.scale,cast(c.is_nullable as varchar) as is_nullable from
   @Count_Distinct m join @ColumnObjectName1 c on  c.ColumnName=m.ColumnName and m.ObjectName=c.ObjectName
  ) t
  
  
  -----------------------------------------------------
  */
DECLARE @temptable3 TABLE (
id INT IDENTITY(1, 1)
,[ColumnName] NVARCHAR(100)
,[ObjectName] NVARCHAR(100)
,DatabaseName NVARCHAR(100)
,DataType VARCHAR(50)
,length INT
)
DECLARE @minMax3 TABLE (
id INT IDENTITY(1, 1)
,value VARCHAR(50)
,count INT
,ColumnName VARCHAR(50)
,ObjectName VARCHAR(50)
,DBName VARCHAR(50)
)
DECLARE @minMax4 TABLE (
id INT IDENTITY(1, 1)
,Minvalue FLOAT
,MaxValue FLOAT
,AvgValue FLOAT
,STDEVValue FLOAT
,DistinctCount INT
,Count INT
,ColumnName NVARCHAR(100) NULL
,ObjectName NVARCHAR(100) NULL
,P FLOAT NULL
,minLen NVARCHAR(100) NULL
,Maxlen NVARCHAR(100) NULL
)

INSERT INTO @temptable3 (
[ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
)
SELECT [ColumnName]
,[ObjectName]
,DatabaseName
,DataType
,length
FROM @BIP_sourcetable
WHERE DataType IN ('int', 'float', 'decimal', 'tinyint', 'smallint')

DECLARE @max3 INT = (
SELECT MAX(id)
FROM @temptable3
)
DECLARE @i3 INT = 1

WHILE @i3 <= @max3
BEGIN
DECLARE @ColumnName3 VARCHAR(50) = (
SELECT [ColumnName]
FROM @temptable3
WHERE id = @i3
)
DECLARE @ObjectName3 VARCHAR(50) = (
SELECT [ObjectName]
FROM @temptable3
WHERE id = @i3
)
DECLARE @DataType3 VARCHAR(50) = (
SELECT DataType
FROM @temptable3
WHERE id = @i3
)
DECLARE @sql3 NVARCHAR(max)
DECLARE @sql4 NVARCHAR(max)

SET @sql3 = ('select ' + @ColumnName3 + ',count(' + @ColumnName3 + ') , ''' + @ColumnName3 + ''',''' + @ObjectName3 + ''',''' + @dbname1 + ''' from ' + @dbname1 + '.' + @ObjectName3 + ' group by ' + @ColumnName3 + '')

INSERT INTO @minMax3 (
value
,count
,ColumnName
,ObjectName
,DBName
)
EXEC (@sql3)

SET @sql4 = (
'select  min(' + @ColumnName3 + '),max(' + @ColumnName3 + '),avg(cast(' + @ColumnName3 + ' as bigint)) ,STDEV(' + @ColumnName3 + '),Count(distinct ' + @ColumnName3 + ' ) as DistinctCount ,
  Count(*) as Count,''' + @columnName3 + ''',''' + @ObjectName3 + ''',(100.0 * SUM(CASE WHEN ' + @ColumnName3 + ' IS NULL THEN 1 ELSE 0 END) / COUNT(*)) as p,Min(len(' + @ColumnName3 + ')),Max(len(' + @ColumnName3 + '))  from ' + @dbname1 + '.' + @ObjectName3 + ''
)

INSERT INTO @minMax4 (
minvalue
,maxvalue
,AvgValue
,STDEVValue
,DistinctCount
,Count
,ColumnName
,ObjectName
,P
,minLen
,Maxlen
)
EXEC (@sql4)

SET @i3 = @i3 + 1
END

DELETE
FROM [Data_Profiling].[dbo].[Data_Profile_ValueDistribution]
WHERE [Database_Name] = @dbname

INSERT INTO [Data_Profiling].[dbo].[Data_Profile_ValueDistribution] (
[Database_Name]
,[Table_Name]
,[Column_Name]
,[Row_Count]
,[NumberOfDistinctValues]
,[Value]
,[Count]
)
SELECT m.DBName
,m.ObjectName
,m.ColumnName
,m1.Count AS [RowCount]
,m1.DistinctCount
,m.value
,m.count
FROM @minMax3 m
JOIN @minMax4 m1 ON m1.ColumnName = m.ColumnName
AND m.ObjectName = m1.ObjectName
END

www.CodeNirvana.in

Powered by Blogger.

Translate

Total Pageviews

Copyright © Journey to MSBI World With Kiran Reddy | Blogger Templates | Designed By Code Nirvana