Server Maintenance Workshop

Server Maintenance Workshop
Part Two: Database Server Brian Bauer, Software Support Specialist

Passageways Database Naming Convention
Entitlement Name used in all Database Names (Production_Portal, Test_Portal, Board_Portal, Community_Portal) _PortalFx - Framework and all Standard Components (Alerts, Framework Islands, Conversations) _PortalFx_Documents – all Database Documents _(modulename) – Each module is stored in its own database Note: For older portals the naming conventions will be different and will not include the entitlement name. The PortalFx database will be named Passageways.

Passageways Database List: Included in Standard Suite
Employee / Board Portal Suite Portal Framework (PortalFx or Passageways) Portal Documents (PortalFx_Documents or Documents) Announcements Calendar Discussions (Only for Pre Installations) Photos Polls Read Receipt Reservations

Passageways Database List: Modules and Featured Apps
Featured Applications Apparel Cart Employee Directory Employee Recognition Invoice Lobby LMS Management Blog Purchase Order Supply Order Test Creator Modules Assigned Tasks Dashboards Expense Reports Forms Help Desk Meeting Packets Products & Services (Products) Vendor Management Note: The naming convention for Featured Applications varies. Sometimes they are included in Passageways_PS or have _PS or CS_ in the database name.

Database Components Tables Data Storage, Table Relationships
Data Indexes, Default values. Views Referenceable queries across multiple tables, usually with functions. Programmability Stored procedures for portal use and functions for queries. – Datediff, Len Storage Index Full Text Catalogs. Security Who has access to the database.

Do NOT change anything in your database. (Expressway Validation)
Things To Remember Do NOT change anything in your database. (Expressway Validation) Backup Databases Regularly Daily! Delete Old Backups / Don’t run out of server space.

On Demand Database Backup
Backup Overview (SQL Server) - MSDN - Microsoft

SQL Data and Logs on Separate Drives
When managing disk space, it can be useful to have SQL database data files and log files on separate drives. This can be managed under the SQL Server properties. After connecting to your SQL server, right-click on the server name in the Object Explorer, and select Properties.

SQL Data and Logs on Separate Drives
Under Database Settings, you can set the default location of the data (.mdb) and log (.ldb) files. Any new databases created will have their files go to these locations. The Passageways portal will only be impacted if the location of the Data or Log file runs out of disk space.

Simple Maintenance Plan Example

SQL Maintenance Plans:
What are they? SQL Maintenance Plans are workflows of tasks required to make sure that your databases are optimized, backed up, and free of inconsistencies! A maintenance plan wizard is available to assist in the creation of maintenance plans, though, often creating plans manually offers more flexibility. Why do we need them? SQL Maintenance Plans are also useful for keeping our data tables small. They allow us to identify large tables, clear out old data, clean out maintenance records and transaction logs and many other things that help keep both our Passageways tables and our master tables in check!

SQL Maintenance Plans: Simple Maintenance Example
From here we will be able to utilize the maintenance plan wizard or manually create a maintenance plan.

In order to send notifications we must first add Operators and configure the necessary information pertaining to each. Before we begin, make sure the SQL Server Agent Service is running and set to Startup Automatically

Right click Maintenance Plan and select New Maintenance Plan. Name the New Maintenance Plan “Simple Maintenance Plan”

SQL Maintenance Plans: Toolbox & Designer
The Toolbox lists the various tasks that can be added to maintenance plans. The Designer will be used to customize the maintenance plan. The Designer Surface (aka Work Area) is used to organize and define the maintenance plan tasks.

Double click to modify the default Subplan_1 Define the subplan properties including a name, description and schedule. We’ll call this one Portal Framework Database Backup. This sub-plan will be used to backup our framework database every day at 3:00 AM.

Drag and drop the Back Up Database Task from the Toolbox into the designer surface. Double click Task to get Properties

SQL Maintenance Plans:
Simple Maintenance Example This is an example of a typical database backup. Backup type: Full, Differential, Transaction Log Database(s): All, System, All user, or Select specific databases. In this case, I selected Portal_Fx and Portal_Fx_Documents Set Backup Compression: Use Default Server Setting, Compress backup, Do not compress backup

Drag and drop the Check Database Integrity Task from the Toolbox into the designer surface. Double-Click the Task Icon to set the properties

Select the Check DB Integrity Task. Drag the green arrow to the Back Up Database Task. You have now created Dependant Tasks by establishing a simple Connection so the tasks will complete in subsequent order.

The type of connections between tasks can be configured as well. Go ahead and double click on the green arrow to get the constraint editor. A typical connection will use the evaluation operation of “Constraint” with a few preset values which include success, failure, and completion. Success – This means that the return value of the operation was successful. Failure – This means that the operation failed to complete or was not successful. Completion – This means that the operation fully completed either with a success or failure.

In addition to checking a simple value based on a constraint, more advanced expressions and operations can be selected, including combinations of constraints and expressions. Generally Expressions are used for validating very specific information such as ensuring that when the previous task completes it is offering up some information that may be validated. The available return values outside of Success, Failure, and Completed are available per task type on MSDN.

Save your new maintenance plan! Ctrl-S or click the Save Document Icon. The maintenance plan will now show under your management folder in the SQL object explorer. There you have it! A simplistic maintenance plan! Just Right-Click and Select Execute to see it in action!

SQL Maintenance Plans: Using the Wizard
If the previous manual way of doing this is intimidating, a wizard exists that guides you through the process of creating a maintenance plan. From the object explorer, right click Maintenance Plans and select Maintenance Plan Wizard.

SQL Maintenance Plans: Using the Wizard
Plan Properties Maintenance Tasks Task Order Task Configuration Report & Log Options Completion Review

Let’s Add Reporting, Logging and Notifications

SQL Maintenance Plans: Reporting, Logging and Notifications
In order to send notifications we must first add Operators and configure the necessary information pertaining to each. Expand the SQL Server Agent node Right click the Operators node and select “New Operator…” Complete the information and click OK.

Now that we have set up an Operator to receive notifications, let’s go back and add some reporting and notifications to our “Simple Maintenance Plan” In your Object Explorer go back to expand Management then Maintenance Plans Double-click, or Right-click the “Simple Maintenance Plan” and select Modify.

Select the Reporting and Logging icon which is kind of hidden between Manage Connections and Servers… You will be presented with the Reporting and Logging dialog.

For my Simple Maintenance Plan I have chosen to… Create a new text file in folder C:\Database Maintenance Reports Send report to Anne Wertz Log extended information. Extended logging will provide additional details when looking back at logs at the cost of additional disk space.

Ctrl-S! Save Frequently! Now you have added simple Reporting and Notifications! Just Right-Click and select Execute to see it in action! Browse to your reports folder to see the results!! If you do not receive an you may need to configure SQL Server Agent Mail to use Database Mail. Resource:

SQL Maintenance Plans: Putting It All Together
Combining all of the skills demonstrated in the previous slides we can add additional tasks to create a more comprehensive workflow that iterates through a daily clean up pattern.

SQL Maintenance Plans: Putting It All Together

Questions?

Historical Data Clean-up

Historical Data Clean-up: Why is it needed?
There are several tables in the Passageways database that keep historical information that may only be pertinent to your organization for a period of time. If the data is never deleted from these tables, they will continue to grow which will increase the size of your database as well as the size of your backups!

Historical Data Clean-up: Using Simple Queries
Let’s take a look at the PWTrafficMonitorEntries table which logs daily every page click made by your users. Using a simple select statement we can view all records sorted by DateTime.

Running just a few more simple queries and our clean-up for this table is complete... For now! 1. Get number of records prior to January 1, 2014 2. Delete all records prior to January 1, 2014

Use DateDiff in your query to find records older than 60 days. DateDiff(Unit of Measurement, Date A, Date B) = Date B – Date A 1. Get the number of records that are older than 60 days 2. Delete all records that are older than 60 days

SQL Maintenance Plans: Tables with Historical Data
Framework Database PWTrafficMonitorEntries PWAlertItems *legacy PWNotification PWSearchTermHistory PWUserWall PWUserWallComment PWUserWallLike Post Announcements PWAnnouncementReadReceipts PWGroupAnnouncements Calendars PWGroupEvents PWGroupEventSignUps Reservations

Data Clean-up Automation
Using SSIS Packages

Data Clean-up Automation: Sample Traffic History Cleanup
Backs up the Traffic History Table Gets the record count for the Traffic History Table Checks if record count <= 500 Checks if record count > 500 Removes records that are older than 60 days.

Data Clean-up Automation: SSIS Packages Overview
SQL Server Business Intelligence Development Studio is a component of SQL Server 2008 R2 that can be included when SQL is installed. SQL Server Integration Services (SSIS) is a project type allow you to create more advanced workflows that can include Maintenance Plan Tasks, Variables, Conditional Connections, SQL Queries and more SSIS Packages can then be scheduled to run regularly in SQL Management Studio

Let’s create a new project called “Traffic History Cleanup” Press the key command “CTRL+SHIFT+N” to pop the dialog for creating a new project. (or use the File Menu) From the templates, select “Integration Services Project” Enter the Project Name, Location and Solution Name. Select OK Note: You may get a wizard when you click OK. Cancel to continue with manual setup.

On the right is the Solution Explorer where you can see the details about the current solution. The default package name is Package.dtsx which is not very descriptive. Let’s rename it to something more easily identifiable. Right click the file “Package.dtsx” and choose rename. Change the name to “TrafficCleanUpPackage.dtsx”. Be sure NOT to lose the extension! When prompted to rename the package object as well, click “Yes”. Ctrl-S! Save Frequently!

Now, in order for this package to interact with our SQL Server Instance, we need to create a connection. At the bottom of the screen is a section called “Connection Managers”. manager Right click this and select “New OLE DB Connection” You will be presented with the Configure Connection Manager. Click the button “New…” now.

Another dialog will pop that allows you to identify server information and provider information. The provider will be dependant on the version of your SQL Instance. Since I am using a SQL 2008 R2 instance, my provider will be Native OLE DB\SQL Server Native Client 10.0. Configure your SQL connection. I have chosen to use Windows Authentication and to connect directly the PortalFX database. Test your connection to ensure that you have selected the right options. Once finished, click “OK”. Select the New Connection, click “OK” again. Your connection should now be in the Connection Manager List.

For this example we need to be able to perform an action based on how many records are in the Traffic History Table. We will get this information from a SQL query later in this exercise. In order to use this information in our SSIS package we will need to have a way to store it. We will define a Variable for this so the data can be used and passed between tasks and connections. Be sure to select the ‘Control Flow’ tab in the designer. Right click in the work area and select “Variables”. The Variables Panel will open to the top, left of the designer

Click the Add Variable button Change the default value to “RowCount” . Ctrl-S! Save Frequently! This variable will be created and will be useable throughout the package. You can have an unlimited number of variables for tracking data throughout your workflows.

Now that we have defined a variable named “RowCount”, we need to run a query to get the number of records in the Traffic History Table, PWTrafficMonitorEntries. Let’s check our Toolbox to see what tasks are available. The Toolbox Button is located all of the way on the left, next to the Variables Panel. Once selected the Variables Panel will be replaced with the Toolbox Panel.

Let’s find the Execute SQL Task and drag it onto the Control Flow work area. Note: If the Execute SQL Task is not available then Right-Click in the Maintenance Plan Task section and ‘Choose Items’ from the menu. Check the Execute SQL Task Item on the SSIS Control Flow Items Tab. Drag and Drop the Execute SQL Task into the work area Double click the Execute SQL Task object to configure Ctrl-S! Save Frequently!

We are able to set a large number of properties for the task but we’re going to cover just the relevant ones at this time. Name / Description ResultSet Property ResultSet Page Connection SQL Statement

Change the Name to “Traffic History Count” so it easily identifies the task. Change the Description to “Return the number of rows in the PWTrafficMonitorEntries table”. Change the Result Set to “Single Row”. Since we are looking to get just the count of records, we are expecting just a single return value from our query. 4. From the connection drop down, select the connection that we created previously.

Now let’s define our SQL Statement. The most important thing about writing this SQL statement is to remember that we will be executing transact SQL. This means that often times it is better to declare variables to pass data throughout the statement. It’s also important to specify the schema object level when interacting with SQL objects. -- Select the number of records in the PWTrafficMonitorEntries table -- and assign it to a local variable. INT -- Local variable of type integer = (Select COUNT(*) from [Production _Portal_PortalFx].[dbo].[PWTrafficMonitorEntries]) -- Sets the Local to the number of rows in this table -- Best practice is to use the dbo schema for defining the object -- Select the value of our local variable and return it in a Column named "TrafficCount" as TrafficCount Tip: Use SQL Management Studio to create and test your SQL Scripts!

Let’s enter the SQL statement from the previous screen into the SQL Query dialog that pops up for our SQL Statement property. Click the Ellipsis on the SQL Statement Property` Copy the SQL Statement in the Enter SQL Query Dialog, Click “OK” INT = (Select COUNT(*) from [Production_PortalFx].[dbo].[PWTrafficMonitorEntries]) as TrafficCount

Finally we have a SSIS Variable, RowCount, that we want to associate with the return value of column, Traffic Count, in our SQL Query. Click on the Result Set page on the left of the screen and click the add button. Click the Add Button. The SSIS Variable, RowCount, will automatically be added Change Result Name from ‘NewResultName’ to ‘TrafficCount’ This will assign the returned value in the column, TrafficCount, to our SSIS variable, RowCount. After setting this, click the OK button on the dialog to save our task.

Ctrl-S! Save Frequently! Before proceeding, we should check to see if our task will execute successfully. To do so, right click the task and “Execute Task”. The task will fire and indicate its success by turning green. If it is red, you have more work to do! Select the progress tab to see the validation output. Hit the Stop button to return to edit mode!

Questions?

Backs up the Traffic History Table Gets the record count for the Traffic History Table Checks if record count <= 500 Checks if record count > 500 Removes records that are older than 60 days.

We have four more tasks to add. Let’s go ahead and move those to our work area and then create our connections. Drag and Drop these tasks to your Work Area Back Up Database Task, Maintenance Cleanup Task, History Cleanup Task, Execute SQL Task Organize the Tasks in Sequential order by moving them around in the work area. (Refer to next slide) Create Connections between each task by dragging the green arrows to connect

Ctrl-S! Save Frequently! Now let’s set the task properties for each task.

Double-click the Back Up Database Task Set the Backup Task Properties as we did when creating SQL Maintenance Plans. Click OK Right-Click Back Up Database Task and “Execute Task” Hit Stop button to return to designer

Double-click the History Cleanup Task Set the History Cleanup Task Properties as we did when creating SQL Maintenance Plans. Click OK Right-Click Back Up Database Task and “Execute Task” Hit Stop button to return to designer

Double-click the Maintenance Cleanup Task Set the Maintenance Cleanup Task Properties as we did when creating SQL Maintenance Plans. Click OK Right-Click Maintenance Cleanup Task and “Execute Task” Hit Stop button to return to designer

Now let’s finish up the Table Cleanup Task! Double-Click the “Execute SQL Task” to get started.

Change the Name to “Delete Traffic History” so it clearly identifies the task. Change the Description to “This task will delete all rows from the PWTrafficMonitorEntries table that are older than 60 days”. Leave the Result Set to “None”. Since we are deleting records, we are not expecting a return value from our query. 4. From the connection drop down, select the connection that we created previously.

Now lets define a SQL Statement that will delete the records from the PWTrafficMonitorEntries table older than 60 days. -- Delete records from the table that are older than 60 days using the DateDiff function -- The number of days can easily be modified based on your organizations needs Delete from [Production_Portal_PortalFx].[dbo].[PWTrafficMonitorEntries] where DATEDIFF (day, DateTime, getdate()) > 60

Let’s enter the SQL statement from the previous screen into the SQL Query dialog that pops up for our SQL Statement property. Click the Ellipsis on the SQL Statement Property` Copy the SQL Statement in the Enter SQL Query Dialog, Click “OK” Delete from [Production_Portal_PortalFx].[dbo].[PWTrafficMonitorEntries] where DATEDIFF (day, DateTime, getdate()) > 60

We now have a linear workflow but we haven’t yet used the RowCount Variable to set a Condition. We only want to run the clean up task if there are more than 500 rows. To do this we can add conditional connections.

Select the “Traffic History Count” Task Drag the unlinked green line from the Traffic History Count to our History Cleanup Task Now you should see two connections from the Traffic History Count task. We want the Delete Traffic History task to happen if there are more than 500 records so let’s work on that connection first.

Double-Click the connection pointing to the Delete Traffic History task.

Switch the evaluation operation to Expression and Constraint. For this we want to both evaluate the data and make sure that the operation succeeded. Leave Value as “Success”. Add an expression using the SSIS variable, RowCount, we created. @[User::RowCount] > 500 4. Click the test button to evaluate our expression. It should give us a success message.

Click OK to accept these constraints. This will ensure that the returned value from our SQL Query must be over the integer 500 before proceeding to the next task. A little fx icon indicates that we are evaluating a variable on this connection. Ctrl-S! Save Frequently!

Now we are ready to add a condition to the other connection. Double-Click the connection pointing to the History Cleanup Task.

Switch the evaluation operation to Expression and Constraint. For this we want to both evaluate the data and make sure that the operation succeeded. Leave Value as “Success”. Add an expression using the SSIS variable, RowCount, we created. @[User::RowCount] <= 500 4. Click the test button to evaluate our expression. It should give us a success message. Ctrl-S! Save Frequently!

Voila! It’s Done! Checks if record count <= 500 Checks if record count > 500 Removes records that are older than 60 days.

Now we can check the whole workflow to make sure it works all together. Hit the Play Button to Start! Check out the Execution Results!

Questions?

Data Clean-up Automation: Schedule and Run!
It is now time to schedule our SSIS Package! Load SQL Server Management Studio Note: It’s OK to leave Business Intelligence Studio Open Navigate to the SQL Server Agent Expand the SQL Server Agent Right Click Jobs and click “New Job…”. A dialog will be displayed that will allow us to start configuring our SSIS package for scheduling. Complete the General properties.

Click the “Steps” page from the left menu and select the “New…” button at the bottom of this page. Provide a Step Name. From the type drop down select “SQL Server Integration Services Package”.

On the General tab change the Package source to “File system”. The Server information should be grayed out. Click the “…” button below “Package” and navigate to the package we created and select it. If you did not take note of the package location, you can go back into your SSIS project and look in the Solution Explorer.

After confirming the job by clicking OK, it is time to create a schedule for the job. Click on the schedule page on the left. Click “New…” at the bottom of the dialog. The schedule dialog should be identical to the SQL Maintenance Plan above. Like before, I have scheduled this to run daily at 3:00 AM.

Finally, let’s create an alert in case something goes wrong with our backup. Click the Alerts page on the left and click the “Add…” button at the bottom of the page. Set the database target to the database defined in our SSIS package and change the severity to 021 – Fatal Error in the Database Processes. This will trigger whenever the database backup fails.

Click the Response page on the left navigation. Here we can choose to notify the operator that we created during the SQL Maintenance Tutorial. That’s it! Go ahead and save your Job. Success!

Questions?

Advanced SSIS Package: Archiving and Renaming Backups
SSIS Packages can utilize a huge range of control flow items. Below are some items that you should look at utilizing in your workflows. Send mail task – The send mail task is great for notifying an individual or group of individuals that your workflow has completed. You will be required to create a new connection manager, like we did for SQL, to your SMTP server for this. This connection can be created from within the task configuration. This task is also great for notifying individuals when tasks failed. Simply trigger an task with a constraint with the constraint value set to “Failure”.

A really important thing to do with SSIS packages is to automate the renaming and movement of backups from a backup server to a storage server. In this part we will look at using 3 tasks together: foreach loop container, empty data flow task and file system task. Start by adding the Foreach Loop Container to your Control Flow.

The next few steps are a bit tricky, but we’ll break down each one. Open up the Foreach Loop Container’s properties by double clicking on it. Click on the Collection menu to the left. We’re going to add an Foreach File Enumerator Expression. In the Enumerator Configuration section, point the folder to your storage server location and for the Files specify *.bak. Finally switch the “Retrieve file name” option to “Name and extension” and select “Traverse subfolders”.

Now we need to create some variables that our foreach loop container will use. All of these variables will be at the scope of the foreach loop container as our other actions will be children of this container. These variables will be named: Archive Path FullArchivePathFileName FullSourcePathFileName MyFileValue SourcePath I’ve placed a temp variable in some of these, however at run time, when iterating many of these filenames will automatically be updated in the loop. Go ahead and create these variables now.

In order for these variables to be useful, we need to add some expressions to them so that they don’t explicitly look for abc or Select the FullSourcePathFileName variable. You should notice that the properties window in the bottom right changes. Select the Expression box and click the “…” button.

A dialog will pop allowing us to write an expression. Enter the following into the Expression box and click evaluate expression: @[User::SourcePath] The output should look like this. Click OK to save this. Now before proceeding, make to change the “EvaluateAsExpression” value to true from false. This will make this variable use the expression instead of the value that we manually entered.

Now we will do the same for FullArchivePathFileName. Follow the preceding steps and this time use the expression: @[User::ArchivePath] + , 1 , - 1 ) + "-" + (DT_STR, 2, 1252) )+ (DT_STR, 4, 1252) )+ , , ) ) This may look a bit confusing, but this takes some basic T-SQL and combines it with our variables. Taking a look at the output makes it make a bit more sense. The real power of this expression is that it names your archive uniquely based on the systems time and date!

Next, open up the properties for the Foreach Loop Container again and this time select the Variable Mappings page. Select the MyFileValue variables for a mapping. This will allow the Foreach Loop Container to own the MyFileValue so that when our rename operation happens later on, it will have a variable to use for the name of the backup.

At this point our Foreach Loop Container is pretty well defined, but it doesn’t have any tasks to loop! Drag a Data Flow Task and a File System Task into the Foreach Loop Container. Then connect the Data Flow Task to the File System Task. Your control flow should look like this: I have renamed the Data Flow Task to “Empty Data Flow Task” to indicate that in this scenario we won’t be using it. In other ETL (extract, transform, load) scenarios, this is where our more complex work would go such as validation, splitting, or compression.

The last part of creating this package is to define the file system task. This task will perform the grunt work. Here we are defining a “Rename file” operation with both destination and source variables. This will result in the Rename file operation also moving our file to the source file path all in one task!

Advanced SSIS Packages Information
At this point, by putting all these steps together we can now place this task into our existing workflow or create a different package all together to contain this loop. If you have done this correct, you can execute at this time. For this example, this is my source folder.

Now, right click foreach loop container and click “Execute Container”. Our container should light up green, our source folder should be empty and our target folder full with renamed backups!

Last, but not least, take a look at the Progress tab to see what the pipeline looks like when executed. After taking a look at this and saving your SSIS package. You can now schedule it and add reporting / logging.

Here is it all put together!

Questions?

Server Maintenance Workshop

Similar presentations

Presentation on theme: "Server Maintenance Workshop"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Server Maintenance Workshop

Similar presentations

Presentation on theme: "Server Maintenance Workshop"— Presentation transcript:

Similar presentations

About project

Feedback