History has its own way to define now civilizations thrived. Most of the cities flourished in the river side and transporting lumber was one of the key activity. Most of the cities like Seattle and many others have this boom and bust life. The idea here was to cut the timber upstream and use the natural flow of rivers to transport to factories downstream using the river. These are classic and wonderful examples of how we typically work with Log-Shipping in SQL Server too. This blog is about Transaction Log Shipping Status report.

The post SQL SERVER – SSMS: Transaction Log Shipping Status Report appeared first on Journey to SQL Authority with Pinal Dave.

A long back I had written a blog on – SQL SERVER – Recycle Error Log – Create New Log file without Server Restart. Recently one of my blog reader emailed me with this question: Hi Pinal, In our SQL Server, we are logging successful logins to ERRORLOG and due...

The post SQL SERVER – How to Increase Number of Errorlog Files appeared first on Journey to SQL Authority with Pinal Dave.

Whenever someone reports some weird error on my blog comments or sends email to know about it, I always ask to share SQL Server ERRORLOG file. There have been many occasions where I need to guide them to find location of ERRORLOG file generated by SQL Server. Most DBA’s are intelligent and know some of these, but this is my try to share my learning about ERRORLOG location.

The post SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location appeared first on Journey to SQL Authority with Pinal Dave.

Building a solution for some sort of disaster site is critical for every organization that thinks about business continuity. One of the proven for ages is a solution around Log Shipping. Though one of the oldest, it is one of the most sought after solution inside SQL Server. With solutions...

The post SQL SERVER – Log Shipping – Sharing Tab Missing in Folder Properties appeared first on Journey to SQL Authority with Pinal Dave.

Every now and then I sit next to DBA to learn some of their tricks. One of this recent learning is what I plan to share here. It is tough to learn everything all by ourselves, so learning is best done by sharing. There are many incidents with SQL cluster...

The post SQL SERVER – Steps to Generate Windows Cluster Log? appeared first on Journey to SQL Authority with Pinal Dave.

The journey for IT started way back in my college days more than a decade back. It was a booming time and it was exciting to see everyone eyeing the Computer Science stream. Almost every other person in those days always wanted to take this new trend. Riding the waves...

The post SQL SERVER – Understanding the Basics of Write Ahead Logging (WAL) Protocol appeared first on Journey to SQL Authority with Pinal Dave.

Readers bump into this blog because it tries to simplify some of the most complex topics into a digestible format. I am a strong believer of this and try to bring some of these topics from time-to-time. Even I find some of these topics confusing, if I can understand them...

The post SQL SERVER – Basic architecture of Transaction Logs appeared first on Journey to SQL Authority with Pinal Dave.

One of my blog readers posted on Facebook asked following question about Huge Transaction Log:

Hi Pinal,
We are using AlwaysOn availability group for our production database. The database is fairly large in size as its around 260 GB. The database transaction log file grows around 2.5gb every day. Over the weekend the DBA team noticed that the Log file for this database bloated up to around 230 GB and then seems to hold steady. While troubleshooting I noted an error that the rebuild Index failed due to "Availability_Replica". Here is the error which we saw in ERRORLOG

The post SQL SERVER – Huge Transaction Log (LDF) for Database in Availability Group appeared first on Journey to SQL Authority with Pinal Dave.

One of my clients asked me more about these messages logged in the SQL Server ERRORLOG. If you are new to SQL Server and don’t know how to reach to ERRORLOG, read below Where is ERRORLOG? Various Ways to Find ERRORLOG Location. In this blog post we will discuss about DBCC TRACEON.

The post SQL SERVER – What is DBCC TRACEON and DBCC TRACEOFF messages in ERRORLOG? appeared first on Journey to SQL Authority with Pinal Dave.

If you are a DBA and if you ever hear about drive space full issues on the SQL Server machine, the first thought to come to your mind would be “huge transaction log file”. Recently I have also fallen into the same trap. One of my clients contacted me and informed that they are running out of drive space on C Drive.

The post SQL SERVER – Drive Space Full Due to MDMP Files appeared first on Journey to SQL Authority with Pinal Dave.

Recovering lost data that was deleted or truncated can be fairly quick and easy, depending on the environment in which the database resides, recovery measures implemented before and after the data loss has occurred, and the tools used for the job. In this blog post we will learn about how to recover truncated or deleted data when a database is (or not) available.

In general, there are three specific scenarios for recovery when the data is lost:

Scenario #1

Full backup has been created before the unintended delete/truncate operations have occurred, and no additional database changes have been performed since. This is by far the easiest data loss to solve, since it can be recovered with the simple restoration of the full database backup, since it will bring the database to the pre-disaster state, and since no additional changes have occurred, the recovery process will be fully valid and completed with the simple restore.

Scenario #2

Same as in the first scenario, the full database backup has been created prior to the deletes/truncates, but these were followed by the additional changes which need to be preserved. If we would restore the full database backup as in the first case, we would lose all of these additional changes, so another solution should be implemented. The most obvious solution is to restore our full database backup as a different database, and then to extract the tables affected by truncate/delete operations, and to import them into the original database.

Even though this solution is easy for smaller databases, in some environments, it is not possible to fully restore full database backups, due to the resources consumed (backup and database size, restoration time, restoration performance impact…). Logical solution here would be to extract those tables for recovery from the full database backup, but this cannot be done via SQL Server mechanisms, nor via SQL Server Management Studio.

In order to extract specific tables from a database backup, we can use ApexSQL Recover, a powerful recovery tool for Microsoft SQL Server which can recover data and structure lost due to delete, drop and truncate operations, recover deleted BLOBs, or extract BLOBs or specific tables directly from database backups without the need to restore them first.

To extract the tables:

1. Install ApexSQL Recover on your workstation (the tool can be used to perform recovery both locally or remotely, so it can be installed directly on the production server to locally complete the recovery/extraction, or it can be installed on the workstation of the user’s preference to remotely connect to the SQL Server and perform recovery from there)
2. Start ApexSQL Recover and choose the option to extract data from a database backup in the main ribbon
   
3. Add database backup file by clicking on the ‘Add file’ button and selecting your full database backup. If you have consecutive transaction log backups which need to be used for table extraction, add them in this step, and check them together with the full database backup
   
4. In the next wizard step, ApexSQL Recover will show the list of tables in the database/backup, and it is recommended to exclude all tables which are of no interest for recovery and to check only tables which were affected by the truncate/delete operations
   
5. In the next step of the wizard, between the choice of saving the recovery script to the .sql file and recovering to a new database, let’s opt for the first option, which enables us to inspect the recovery script, or to execute the recovery script against the production database to complete the recovery. We can also specify the script location.
   
6. In the final step, we should select the output type. Since we want to recover only data lost due to truncate/delete operations, choose the ‘Extract data only’ option and click next
   
7. After ApexSQL Recover completes processing, the application will inform the user on the recovery outcome, and users can now inspect the script directly from the application internal editor, or open it manually in a SQL Server Management studio or similar tool, from where it should be executed against the production database to complete the recovery process
   

Scenario #3

A different scenario, and by far the worst one, is when a disastrous truncate/delete occurs, followed by additional important transactions which we also do not want to lose, but there is no full database backup to use as the recovery source. Since we do not have a database backup, restoring it, or extracting table data with ApexSQL Recover is not an option.

Luckily, we can use ApexSQL Recover to perform the recovery of truncated/deleted data, since it does not require full database backup in the recovery process, but instead utilizes information in the LDF and MDF files to rollback unintended changes. With this in mind, knowing the internal mechanisms of the SQL Server itself, prior to engaging the actual recovery, due to the fact that SQL Server overwrites MDF file information frequently, it is important to ensure that the information within is kept safe, and not overwritten by the upcoming traffic.

In general, here are the immediate steps to take once the disaster is detected:

1. Change database mode to ‘Read only’ to prevent overwriting of the information in the MDF file
2. Take the database offline and make copies of database LDF and MDF files, then take the database back online – these copies can then be used to create additional copies of the database which can be used to read recovery information from, and enable the production database to continue to receive changes while the recovery process is prepared and performed.

Even though both of these measures will mean that your database will have some downtime, it is important to take these precautionary steps whenever possible to secure the highest change of successful recovery.

As said above, we’ll use ApexSQL Recover to connect to the database affected by unintended truncate/delete operations and generate a recovery script. Here is how:

1. Start ApexSQL Recover and choose to recover ‘Deleted data’ or ‘Truncated data’, depending on the operation which you need to recover from – the remaining steps of the wizard will be the same regardless of this choice
   
2. In the next step, provide the connection details to your SQL Server instance, provide connection credentials and choose the database for recovery
   
3. In the next step, we can go with any of the three options, depending on our case:
   1. ‘Help me decide’ option will guide the user through a quick series of questions in order to provide the most appropriate resources
   2. ‘Add transaction logs’ option should be used if we are creating transaction log backups so we can use them in addition to the MDF and LDF files as the recovery source
   3. ‘No additional transaction logs are available’ option should be chosen if our transaction log is not being backup-ed
      
4. Regardless of the choice in the previous step, once we proceed, we will arrive to the step which is used to determine the recovery window – a time frame which will be considered for the recovery. If we do have a knowledge of the time frame where a disaster has occurred, we should specify the time frame, as close as possible, which will result in faster processing, and in a more accurate recovery (additional delete/truncate operations may have occurred close to the disastrous once, so we will want to exclude those from the recovery process by using proper date/time filters)
   
5. The next step allows us to further filter-out the recovery and narrow it down to specific tables, so it is recommended to include only tables affected by disastrous truncates/deletes
   
6. Same as when we were extracting table information in the scenario #2, we will choose to create a recovery script, and once the processing is finished, our recovery script can be inspected and finally executed against the production database to complete the recovery process
   

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – How to Recover Truncated or Deleted Data When a Database Backup is or is not Available

There are many DBAs who create jobs in SQL Server to recycle ERRORLOG files every midnight. They also increase the files from 6 to a bigger number so that they have data for more than 6 days.

Here is the blog where I wrote about sp_cycle_errorlog. SQL SERVER – Recycle Error Log – Create New Log file without a Server Restart

You can read comments on above blog and see various usage scenarios.

One of the blog reader quoted my blog and said that I am unable to recycle ERRORLOG file using sp_cycle_errorlog command. I asked to share the ERRORLOG file content while running the command. Here is what we see in ERRORLOG.

2016-09-09 08:24:37.45 spid70 Attempting to cycle error log. This is an informational message only; no user action is required.
2016-09-09 08:24:37.46 spid70 Error: 17049, Severity: 16, State: 1.
2016-09-09 08:24:37.46 spid70 Unable to cycle error log file from ‘E:\MSSQL10_50.INSTANCE1\MSSQL\Log\ERRORLOG.5’ to ‘E:\MSSQL10_50.INSTANCE1\MSSQL\Log\ERRORLOG.6’ due to OS error ‘1392(The file or directory is corrupted and unreadable.)’. A process outside of SQL Server may be preventing SQL Server from reading the files. As a result, errorlog entries may be lost and it may not be possible to view some SQL Server errorlogs. Make sure no other processes have locked the file with write-only access.”

I asked to open file in SSMS and he informed that he is getting error.

.Net SqlClient Data Provider Unicode file expected.

All of these point to corruption to the file in such a way that SQL is not able to read them correctly. Strangely, they were able to open via notepad.

SOLUTION

I have asked them to make sure that there is no corruption on the E drive. Since we were able to read the files via notepad, I told them to move “bad” file to some other location and then try.

We moved all ERRORLOG.* files to a different folder and after that recycle worked well.

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Recycle ERRORLOG Using sp_cycle_errorlog

Once, while doing a load testing with my client, I was shown below error while taking log backups on secondary replica.

Msg 35295, Level 16, State 1, Line 1
Log backup for database “MyAppDB” on a secondary replica failed because the last backup LSN (0x0016ff11:00032258:0001) from the primary database is greater than the current local redo LSN (0x0016ffec:0006ee28:0001). No log records need to be backed up at this time. Retry the log-backup operation later.
Msg 3013, Level 16, State 1, Line 1
BACKUP LOG is terminating abnormally.

This error was interesting because as per documentation we can take backup on the secondary replica in AlwaysOn configuration. I quickly checked AlwaysOn Dashboard from Management studio, all were looking green and databases were synchronized. Then I checked transaction log file (LDF) size and it was very big which made me think in different directions. Went to dashboard again and added Redo queue size. We found that REDO thread was blocked by a user query. As per Sys.dm_exec_requests we found that SPID 28 (which was showing DB STARTUP) was blocked by a user SPID 255. Then we used conventional way and found that SPID 255 was running SELECT INTO query for many hours. This was from a job which had a complex join with huge tables.

WORKAROUND/SOLUTION

If you find the same error while taking log backups on secondary replica, then check REDO THREAD and check if its blocked. Based on criticality you need to take a decision to KILL the SPID which is blocking system SPID.

As soon as we killed user query, the “Recovery Queue” for that database started coming down. Once it came down to zero, we could take a transaction log backup on the secondary.

Have you seen REDO blocked in production environment? Do you monitor this by monitoring tool?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Msg 35295, Level 16 – Log Backup For Database ‘DBName’ on a Secondary Replica Failed

One of my clients contacted me and informed that they have a node production clustered server and they recently noticed that the nightly error log had stopped cycling ERRORLOGs. When we attempted to run sp_cycle_errorlog we get the following:

Msg 17049, Level 16, State 1, Procedure sp_cycle_errorlog, Line 9
Unable to cycle error log file from ‘G:\SQL_DATAMNT\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG.10’ to ‘G:\SQL_DATAMNT\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG.11’ due to OS error ‘1392(failed to retrieve text for this error. Reason: 15105)’. A process outside of SQL Server may be preventing SQL Server from reading the files. As a result, errorlog entries may be lost and it may not be possible to view some SQL Server errorlogs. Make sure no other processes have locked the file with write-only access.”
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Above error has one meaningful information which is not very clearly visible. OS Error 1392. What does it mean?

As shown above, the error means – The file or directory is corrupted and unreadable.

When we try opening ERRORLOG.10 we got exactly the same error. What is the solution now?

WHY ERROR?

When SQL Server recycles the ERRORLOG, it will rename all the previous ERRORLOG.n to ERRORLOG.(n+1), and then create a new ERRORLOG as the most current one. If the rename process fails, the recycle of ERRORLOG cannot complete.

WORKAROUN/SOLUTION

Since we can’t use that file and in this case even deletion of the file was also not happening. We changed -e parameter in configuration manager to use the new path.

Then we failed over the SQL cluster instance to another node. We verified that the SQL server can come online and ERRORLOG generated at the new location. Then we can recycle command again and it worked!

CONCLUSION

Note that this is a serious error. I told my client that he was lucky that it happened with ERRORLOG file. Imagine if it happens with database related MDF or LDF file. They immediately engaged a hardware vendor for the disk sanity check.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to recycle Errorlog – sp_cycle_errorlog – OS error 1392

This blog is a result of a Skype conversation with one of my clients. The issue, for which they contacted me, was that they were not able to stat SQL Service due to database may not be activated. I asked to share SQL Server ERRORLOG files using below blog SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here is the information which I saw.

2017-04-02 16:38:02.40 spid11s Error: 913, Severity: 16, State: 6.
2017-04-02 16:38:02.40 spid11s Could not find database ID 3. Database may not be activated yet or may be in transition. Reissue the query once the database is available. If you do not think this error is due to a database that is transitioning its state and this error continues to occur, contact your primary support provider. Please have available for review the Microsoft SQL Server error log and any additional information relevant to the circumstances when the error occurred.

The above message is scary because it says that it can’t fine Database ID = 3. If you look at documentation, it is the ID for model database. We know that model is a system database which is needed for SQL to start and any issues with this database would cause SQL startup failures. In my case, the client didn’t know how it was detached. Now, the next step is what we should do?  There were NO backups available. We had MDF and LDF files. So, I tried below.

Solution / Workaround:

• I started the SQL Instance using trace flag 3608, which is documented in books online. Then we tried to attach the model database and got the below error:

Create Database failed
Error: 33401, Severity: 16, FILESTREAM database options cannot be set on system databases such as model.

• We tried picking model database files from different servers, but no luck.
• Since there was no backup, we were not able to try to restore.

Finally, I have asked them to rebuild the system databases by keeping the copy of MDF and LDF files for all databases as a backup.

Setup /QUIET /ACTION=REBUILDDATABASE /INSTANCENAME=MSSQLSERVER /SQLSYSADMINACCOUNTS=”BUILTIN\Administrators” /SAPWD= AVeryStrongP@$$word123 /SQLCOLLATION= “SQL_Latin1_General_CP1_CI_AS”

Once above command was finished, it was as good as fresh install of SQL Server. Later we attached all user databases and created all system objects.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: Error: 913, Severity: 16 – Could Not Find Database ID 3. Database May Not be Activated Yet or May be in Transition – SQL Service

During one of my consulting engagement about Always On availability group configuration, I found that listener network name was not coming online. As a rule of thumb, I always generate cluster log for any error related to cluster. SQL SERVER – Steps to Generate Windows Cluster Log?

Here is the information in the cluster log. I have removed the timestamp column to make it readable.

DBG [RHS] Resource SQLAUTH-PRODAG_SQLAUTH-LISTENER called SetResourceStatusEx: checkpoint 2. Old state OnlinePending, new state OnlinePending, AppSpErrorCode 0, Flags 0, nores=false
ERR [RES] Network Name: [NNLIB] Unable to add server name SQLAUTH-LISTENER to transport \Device\NetBt_If4, status 2
INFO [RES] Network Name : Netbios: Registered Name: SQLAUTH-LISTENER into Netbios (Type: Singleton), result: 2 (0 transports registered)
INFO [RES] Network Name : Netbios: Slow Operation, FinishWithReply: 2
INFO [RES] Network Name : Netbios: End of Slow Operation, state: Initializing/Idle, prevWorkState: Idle
INFO [RES] Network Name: Agent: OnInitializeReply, Failure on (1dac30da-a46a-4c44-923b-d428c6329d6d,Netbios): 2

I look for ERR in cluster log and here is the error which we can see above.

Unable to add server name SQLAUTH-LISTENER to transport \Device\NetBt_If4, status 2

Status 2 indicates: The system cannot find the file specified”. My search for “NetBt” was pointing to issue with “NetBIOS over TCPIP” setting

WORKAROUND/SOLUTION

When we checked Network Name resource properties, we saw “The system cannot find the file specified.” for NetBIOS status. Now to clear that, I looked at IP address resource and unchecked NetBIOS related setting as shown below.

If you want to change that setting a NIC level, then you can follow my earlier blog.

SQL SERVER – FIX Error – Cluster Network Name showing NETBIOS status as “The system cannot find the file specified”

After fixing above, we were able to bring listener online.

Let me know if you have ever faced similar error with windows cluster log. I would like to know your opinion and workaround as well. Remember sharing is the caring.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to Add Server Name SQLAUTH-LISTENER to Transport Device NetBt_If4 Status 2 – Windows Cluster

There have been many situations where rather than performance tuning, I am contacted for an installation issue. Due to my knowledge by looking at logs and pinpointing at the error message, I am able to fix them as well. In this blog post we will learn how to fix Error Installing Microsoft Visual Studio 2010 Shell.

Here is another blog which is related to patching issue which I was asked to fix recently. In the UI, the failure was caused while installing Visual Studio 2010 Shell. I looked at log pointed which was VC10Redist_Cpu64_1.log

Here is the content of the file.

******* RunEngine:
******* Product: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi
******* Action:
******* CommandLine: **********
MSI (s) (A8:84) [13:58:01:399]: Note: 1: 2203 2: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi 3: -2147287037
MSI (s) (A8:84) [13:58:01:399]: MainEngineThread is returning 3
MSI (s) (A8:64) [13:58:01:415]: User policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Machine policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (A8:64) [13:58:01:415]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (A8:64) [13:58:01:415]: Decrementing counter to disable shutdown. If counter
MSI (c) (90:CC) [13:58:01:415]: MainEngineThread is returning 3

This was the only information in the log. If we have a look at the initial stage of the setup file, we see an error related to C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

I checked the file and it was not there in the location mentioned above. We were not sure how it was missing. May be someone deleted the file due to space issue on C drive? For the path, it looks like the SQL setup caches the media locally for un-installation purpose?

WORKAROUND/SOLUTION

We download patch from KB3045318 again and looked for the file. It was found under VC10SP1\x64\vc_red.msi  folder. The fix the above issue was that we had to copy vc_red.msi directly into VC10SP1 so that the final path looks like — C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Error Installing Microsoft Visual Studio 2010 Shell

While I work on performance tuning most of the time, I do provide consultancy on issues which can be resolved quickly. These are the engagement where I get a lot of content for my blogs as well. Let us learn about the error related to redo log record.

One of my clients contacted me when they were not able to start their SQL Service. Without wasting any time, I got engaged with them and ask to share Errorlog to see the cause. Below is the error which I could see from the ERRORLOG.

2017-05-22 04:22:55.97 spid7s Starting up database ‘master’.
2017-05-22 04:22:55.98 spid7s Error: 3456, Severity: 21, State: 1.
2017-05-22 04:22:55.98 spid7s Could not redo log record (2579:456:5), for transaction ID (0:151685), on page (1:375), database ‘master’ (database ID 1). Page: LSN = (2579:424:5), type = 1. Log: OpCode = 4, context 2, PrevPageLSN: (2579:368:3). Restore from a backup of the database, or repair the database.

Since the master database was having corruption, there were not many choices left. The error message clearly says what we need to do. The message generally comes when there is a corruption in the transaction log file of the database. In our case, it was master database, which is one of the system database and due to that SQL was not getting started.

WORKAROUND/SOLUTION

To come out of the situation, below are the options.

• Restore from backup
• Rebuild System DBs
• Reinstall SQL Server

If we choose to restore from backup, then first we need to rebuild system databases from setup.exe and then perform disaster recovery procedures. Below is the command which can be used to rebuild system databases. We have put instance name as MSSQLServer because it was a default instance. For named instance, we need to use instance name. Make sure to take backup of existing MDF and LDF at a safe location before rebuilding.

setup /ACTION=REBUILDDATABASE /QUIET /INSTANCENAME=MSSQLSERVER /SQLSYSADMINACCOUNTS=”Domain\LoginName” /SAPWD=P@ssw0rd

It completed successfully and SQL was up and running. Now, this SQL server was as fresh as a new installation with no information about the earlier configuration of SQL Server. My client was having a hard luck as they were not having a backup of any system databases. All they had was MDF and LDF file of their user database. We tried to attach the user database but got the below error.

CREATE FILE encountered operating system error 5(Access is denied.) while attempting to open or create the physical file ‘S:\MSSQL2016\MSSQL14.MSSQLSERVER\MSSQL\Data\AX_PROD.mdf’. (Microsoft SQL Server, Error: 5123)

Then we went to the file properties and saw that this file was orphaned file as there was no owner. We provided SQL Startup account as the owner of this file. Once permission was set correctly, we were able to attach the database files and bring user database online.

Later, we stopped SQL Service and replaced database files for MSDB and model database and started SQL again. This time they were lucky and MSDB was clean so all information in MSDB came back. They were able to see the jobs etc. There were not many logins in the master database so they were able to recreate them and application was back online.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Error: 3456, Severity: 21 – Could not redo log record (xx:xx:xx), for transaction ID (x:xx), on page (x:xx), database ‘master’

One of my clients wanted to secure their SQL Server. One of the rules was that wanted to avoid running SQL Server service with an account which is part of local “Administrators” group. They wanted to provide the minimal permissions needed for the SQL server service account for SQL Server to function.

Below are the respective articles for each version that outline the permissions assigned to the service accounts:

Configure Windows Service Accounts and Permissions (SQL Server 2016 and later)
https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-windows-service-accounts-and-permissions

Configure Windows Service Accounts and Permissions (SQL Server 2014)
https://msdn.microsoft.com/en-US/library/ms143504(SQL.120).aspx#Windows

Configure Windows Service Accounts and Permissions (SQL Server 2012)
https://msdn.microsoft.com/en-US/library/ms143504(SQL.110).aspx#Windows

Setting Up Windows Service Accounts (SQL Server 2008 R2)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.105).aspx#Review_NT_rights

Setting Up Windows Service Accounts (SQL Server 2008)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.100).aspx#Review_NT_rights

Setting Up Windows Service Accounts (SQL Server 2005)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.90).aspx#Review_NT_rights

When I spoke to my client, they already removed service account from the Administrators group. As per article, we have permissions in security policy, but still we were not able to start SQL Service. By looking further into the Event Viewer, under Application Log we see the error :

“initerrlog: Could not open error log file ”. Operating system error = 3(The system cannot find the path specified.)”.

SQL SERVER – Event 17058 – initerrlog: Could not Open Error Log File

If we provide incorrect path, we generally see below message.

initerrlog: Could not open error log file ‘C:\BadPath\Errorlog.txt’. Operating system error = 3(The system cannot find the path specified.).”

Notice the blank path in an error message.  Later, we used one of my favorite tools called as Process Monitor and ran the tool to look for registry read activity. We knew that the service was queried for the for the start-up parameters which are saved in the registry and we wanted to check how it does it. We saw we were getting Access Denied for the Account: Americas\SAPPROD_SQLSvc on the registry hive:

HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQLServer\Parameters

Seeing that, we went to the registry hive and checked if the service account had permission and found that the user did not have permission, we also saw that the there was a group, SQLServerMSSQLUser$MPNA01$MSSQLSERVER

WORKAORUND/SOLUTION

We were able to fix the issue by adding Service Account Americas\SAPPROD_SQLSvc to the group SQLServerMSSQLUser$MPNA01$MSSQLSERVER

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Start SQL When Service Account is Removed From Local Administrators Group. Why?

Many companies do hardening of servers to make sure they are protected from any known vulnerabilities. It is important to test all the application in test server before doing hardening. Let us learn about how to fix Eventlog Service not starting.

Recently, one of my existing clients contacted me and informed that they are not able to start SQL Server Agent Service after running a hardening script. Without wasting much time, I joined the session with them and started looking at the logs.

I asked to start SQL Server Agent from configuration manager. It failed with an unhelpful error message

The request failed or the service did not respond in a timely fashion. Consult the event log or other applicable error logs for details.

When I checked the LOG folder, there was no SQLAgent.out generated. I thought of checking the event viewer and found below the message.

But is that related to SQL Server Agent startup issue? I ran the SQLAgent executable in console mode using below command

SQLAGENT.EXE" -i SQL2016 -c -v

This failed with an error:

The EventLog service has not been started
2017-08-03 09:24:45 – ? [098] SQLServerAgent terminated (normally)

When I checked “Windows Event Log” Service, it was stopped, and I was unable to start it. I was getting access denied error when trying to start.

Windows could not start the Windows Event Log service on Local Computer. Error 5: Access is denied.

I used Process Monitor tool and found that we had “Access Denied” on C:\Windows\System32\winevt\Logs\System.evtx file. This is the System Event Log. When we checked the properties, we found “Read-only” was checked which was not the case with other machines.

As soon as we removed the checkbox, we were able to start SQL Server Agent service.

Have you ever encountered such issues due to hardening? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Agent Not Starting. The EventLog Service has Not Been Started