Been a while since I posted due to being busy, however I did come aross an issue for a customer which I felt was worth posting.
Firstly, the issue came about when the client noticed all of a sudden that some custom SQL Reports were not loading. Quick initial investigation showed that the ProjectWorkspaceInternalHRef column in the database had no entries at all for the site URLs, so thus the parameter in the report which displayed project names, and brought back site information, did not work as it used this very column.
Checking the Project Sites section in PWA’s Server Settings area, showed the URLs just fine, and sure enough the sites loaded fine. We could add risks and issues in them, link them to tasks etc. However. There was behaviour problems other than this column in the ‘MSP_EpmProject_UserView’ that we found.
Trying to sychronize these sites in PWA that show, threw back this error in the Queue;
GeneralQueueJobFailed (26000) – SynchronizeMembershipForWssSite.SynchronizeMembershipForWssSiteMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’c0118dc8-ef37-46d8-a1ce-db91e6dec178′ JobUID=’f08e920c-195e-4f99-ab81-0769c92071c5′ ComputerName=’SERVERNAME’ GroupType=’SynchronizeMembershipForWssSite’ MessageType=’SynchronizeMembershipForWssSiteMessage’ MessageId=’1′ Stage=”. For more details, check the ULS logs on machine SERVERNAME for entries with JobUID f08e920c-195e-4f99-ab81-0769c92071c5.
Trying to run a reporting database refresh at this stage put this error in the Queue;
Reporting message processor failed:
ReportingRDBRefreshMessageFailed (24023) – RDB area: Epm, error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep time: 00:05:00. Details: id=’24023′ name=’ReportingRDBRefreshMessageFailed’ uid=’0e7e5d1a-c8d8-4386-b545-60750629b023′ QueueMessageBody=’One of the stages of the Refresh operation failed’ Error=’RDB area: Epm, error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep time: 00:05:00′.
GeneralQueueJobFailed (26000) – ReportingRefresh.ReportRefreshMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’a85fd82d-660f-4c3d-a7f3-ce212c851df7′ JobUID=’baadba82-b62e-4c51-a82e-053c0ed2b6e0′ ComputerName=’SERVERNAME’ GroupType=’ReportingRefresh’ MessageType=’ReportRefreshMessage’ MessageId=’1′ Stage=”. For more details, check the ULS logs on machine SERVERNAME for entries with JobUID baadba82-b62e-4c51-a82e-053c0ed2b6e0.
Migrating the databases to a ‘Test’ environment, allowed the problem to be replicated, with one subtle difference. The Project Sites did NOT show in Server Settings > Project Sites. The sites did still however exist in View All Site Content. Taking the URL, and adding it into the Project Sites section threw this error in a dialog box;
The Web site does not exist or is not configured for Project Server. Enter a Web site that has been extended with a Project Server compatible template.
The provisioning of the test instance did also allow me to create NEW project sites, and these placed URLSs into the ProjectWorkspaceInternalHRef column. Existing sites still maintained the same problem however. So, at this point I went to the original problematic Project Server instance, and into Central Admin and re-provisioned the PWA Instance in the Managed Service Application (not changing any settings). New sites could be created once again.
Now I decided to run Bulk Update Project Sites in Server Settings, which threw this error in the Project Server queue;
Web does not exist:
WSSWebDoesNotExist (16405). Details: id=’16405′ name=’WSSWebDoesNotExist’ uid=’7c7fbd82-5a04-4ab3-8f2a-92b914cef946′ projectUID=’31ee44ba-11ed-4e10-b43e-fd9823f136f2′ wssFullUrl=’/PWA/Birmingham Courtyard and Security Improvements’.
GeneralQueueJobFailed (26000) – UpdateProjectSitePath.UpdateProjectSitePathMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’d5124923-52f1-4430-b253-55e2d16bc300′ JobUID=’61878907-2ada-4b2f-8adf-d1057d1ecbf5′ ComputerName=’SERVERNAME’ GroupType=’UpdateProjectSitePath’ MessageType=’UpdateProjectSitePathMessage’ MessageId=’1′ Stage=”. For more details, check the ULS logs on machine SERVERNAME for entries with JobUID 61878907-2ada-4b2f-8adf-d1057d1ecbf5.
Publishing existing sites (the ones without the URL’s in the ProjectWorkspaceInternalHRef view) produced this error;
Given project already has a web associated with it:
WSSProjectAlreadyHasSpWeb (16404). Details: id=’16404′ name=’WSSProjectAlreadyHasSpWeb’ uid=’ba6b06af-8bfa-4d0f-a47e-34922b59e2bd’ projectUID=’873b9d7a-bab9-4be0-9690-d44330d46387′.
GeneralQueueJobFailed (26000) – CreateWssSite.CreateWssSiteMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’bbef0d29-49e5-44ea-b982-3cfbc67e292c’ JobUID=’bff918d6-b6c3-480c-8291-64b92ceed0c4′ ComputerName=’SERVERNAME’ GroupType=’CreateWssSite’ MessageType=’CreateWssSiteMessage’ MessageId=’1′ Stage=”. For more details, check the ULS logs on machine SERVERNAME for entries with JobUID bff918d6-b6c3-480c-8291-64b92ceed0c4.
Clearly they do have a website… We can navigate to it. At this point we even tried resaving ALL custom fields and lookup tables followed by a RDB refresh, still same problems…
Final Steps and Solution
So, at this point, myself and a colleage went on the clients site to investiage the issue as I was using a cranky connection in, and it was making any progress difficult. Most of the day was spent going over similar steps so to skip around this, a solution did finally come about. There was another error when running the RDB refresh, the same as found in my post for another client https://spfactoryuk.wordpress.com/2012/10/16/foreign-key-constraint-fk_msp_epmtaskbaseline_projectuid_taskuid-error-in-project-server-queue/ – basically orphaned basline guids. We run the scripts to clean it up (only one project was impacted), and then the reporting database rebuilt, so progress at last…
Only thing following this, was there was no Project Sites in the ProjectWorkspaceInternalHRef view still, but ok, we need to publish the projects… which was not quite enough. We actually had to delete the PWA instance and Web Application (NOT THE DATABASES THOUGH!) then recreate them linking back to the databases.
Once the PWA site was back up, we had a Project Sites section, with no url’s listed, just like the test environment had when we restored there. So from here, we did a RDB refresh with success, then simply added the URL’s (the sites were still in View All Site Content) back in, THEN published the projects and hurrah things were happy again!
I did also raise a TechNet post, thanks to the help of Hrishi and Paul, the guidance did prove I wasn’t going mad for the most part! Link here;
Also, the client was February 2012 CU, which is known to hold the orphan baseline issue. I have recommended they go for December 2012 to avoid it happening again.