Today’s
article is based on the numbers of problems that I faced while configuring fast
search. This is the first time I have done this configuration but there were
string of problems I encountered while resolving the issue. I would like to
share the complete troubleshooting steps so that your time will be saved, issue
will be resolved, you can meet your deadline in case of any, reduce the
frustration J
3. Verified that FAST Service Account is part of FASTSearchAdministrators and FASTSearchKeywordAdministrators group
5. We Provide Full Control for Machine Keys in the following location on FAST Server.
(location" “Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\ Machine Keys”)
6. After this we used a new browser and browsed to the site.
7. When examined SharePoint ULS, was seeing error below,
SP attributes:
SharePoint
Server 2010 with recent CU
Fast
Search Server 2010
Problem
description:
I
have configured the Fast Search and added the content sources. I was able to
crawl but found out Search results are not coming / working.
Error Message:
"Internal Server error exception" while searching
from SharePoint sites.
Troubleshooting
done:
1. Verified certificate installation and configuration on both FAST and SharePoint servers
1. Verified certificate installation and configuration on both FAST and SharePoint servers
2. Examined Service
application proxy, TCP Port status and IPV6 status of FAST and SharePoint
servers
3. Verified that FAST Service Account is part of FASTSearchAdministrators and FASTSearchKeywordAdministrators group
Ref: http://technet.microsoft.com/en-us/library/ff599535%28v=office.14%29.aspx#BKMK_Useraccounts
4. While troubleshooting FAST service and SharePoint Search Service status identified that SAMWorker process at FAST ADMIN Node was in DEAD state.
4. While troubleshooting FAST service and SharePoint Search Service status identified that SAMWorker process at FAST ADMIN Node was in DEAD state.
About SAMWorker:
Generates
a security filter for each user to trim down the query results returned to a
user.
Inputs End user claims tokens from the
QRProxyService.
User
security filter requests from the QRServer.
Accepts
configuration changes from the SAM Admin.
Outputs
Generates a user’s security filter which is joined with the user’s query in the
QRServer to do security trimming.
Dependencies:
LogService, SAM Admin Service, QRServer, QRServerProxy.
5. We Provide Full Control for Machine Keys in the following location on FAST Server.
(location" “Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\ Machine Keys”)
6. After this we used a new browser and browsed to the site.
Got
the error "The search request was unable to
connect to the Search Service"
7. When examined SharePoint ULS, was seeing error below,
02/24/2014
09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hr Monitorable
FastSearchGateway::GetQueryResult--Communication Exception searching on QRServer:
http://SharePoint.amolgalaxy:13287. Putting qrserver on the exclude list for a
while. Exception: System.ServiceModel.FaultException: The server was unable to
process the request due to an internal error. For more information about the error, either
turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or
from the <serviceDebug> configuration behavior) on the server in order to
send the exception information back to the client, or turn on tracing as per
the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace
logs. Server stack trace: at
System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply,
MessageFault fault, String action, MessageVersion version, FaultConverter
faultConverter) at
System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime
operation, ProxyRpc& rpc) at
System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway,
ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan
timeout) at
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message) Exception rethrown at
[0]: at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg,
IMessage retMsg) at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at
IProxyRemote.ProcessRequest(String request)
at Microsoft.SharePoint.Search.Extended.Query.Http.WcfSearchEngine.GetSearchResponse(IQuery
query, String searchUrlPath, String urlQueryStr, Int32 timeout) at
Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.SearchInternal(IQuery
query, IResultParser parser, String urlStr)
at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.Search(IQuery
query) at
Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery
query). f72be699-6dae-42d0-8bd6-501f61958e3d
02/24/2014
09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hk
Monitorable AbstractSearchGateway::Search--Exception when searching for
sharepoint site:"http://amolgalaxy/sites/admin-apps". Ex:
Microsoft.Office.Server.Search.Query.SearchServiceNotFoundException: Service
fault communicating with query service http://SharePoint.amolgalaxy:13287;
The server was unable to process the request due to an internal error. For more information about the error, either
turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or
from the <serviceDebug> configuration behavior) on the server in order to
send the exception information back to the client, or turn on tracing as per
the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace
logs. at
Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery
query) at
Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.ExecuteSearch(SearchRequest
request, String queryString) at
Microsoft.Office.Server.Search.Query.Gateway.AbstractSearchGateway.Search(SearchRequest
request) f72be699-6dae-42d0-8bd6-501f61958e3d
8. We confirmed that
the Proxy for Search is up and running by executing
Get-SPServiceApplicationProxy
-
Checked the proxy setting on the IE and was not configured.
-
Tried to start the SamWorker (which is dead) but it failed to start
-
In the event viewer found the following error
Log
Name: Application
Source: FASTSearchSAMWorker
Date:
Event
ID: 0
Task
Category: None
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
Service
cannot be started. System.InvalidOperationException: Error Starting up. Unable to initialize Startup Object
Microsoft.SharePoint.Search.Extended.Security.Base.WCF.WCFServiceManager,Microsoft.SharePoint.Search.Extended.Security.Base"
at
Microsoft.SharePoint.Search.Extended.Security.WorkerService.WorkerService.OnStart(String[]
args)
at
System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
9. Followed steps below in an attempt to resolve this error,
- Stopped the Fast search service.
- Modified the Fast folder (%fastsearch%) permissions to add
'FASTSearchAdministrators' group members one by one, with full control.
- Set child objects to inherit permissions
- Removed the tilda files begining with ‘~’ from %fastsearch%
folder
- Started the Fast services for SharePoint
It
did not help resolve the issue. SamWorker remained in stopped state.
NCTRL
Status showing SAM worker 'dead'
state.
Services.msc> FAST Search SAM Worker
Service is not started. Manually startign it is not helping to get sam worker running.
When
ran commands below in given order,
-nctrl stop samworker
-nctrl stop samadmin
-nctrl start samadmin
-nctrl start samworker
The SAMWorker process gets in to running state and goes back to dead
state in 2-3 minutes.
You
are not able to get any search results in the SharePoint farm. Crawling and
indexing works as expected.
This is a new deployment and the issue explained above was
identified during the deployment phase.
-Checked %FASTSEARCH%/var/logs/syslog/ found only 'authorization-admin_amolgalaxy.txt' . So,
this system never has SAMworker running.
If so, we should see at least authorization-worker_ amolgalaxy.txt in
same folder.
-Tried to set FAST logging from warning to debug
Ran commands below
at customer fast server,
-Set-FASTSearchSecurityLogLevel -DefaultLogLevel
debug --> Got error "cannot
set log level."
-Set-FASTSearchSecurityLogLevel
-WarningNameSpaceLogLevel debug -->
Got error "cannot set log level.”
Checked the config file in location Config File(s) Location and verified that they are good by comparing
with a good FAST server's configuration files.
For
SAM ADMIN:
%FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.AdminService.exe.config
%FASTSEARCH%/components/sam/admin and
For SAM WORKER
%FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.workerService.exe.config
%FASTSEARCH%/components/sam/worker
-Followed http://support.microsoft.com/kb/2656160
and ran "psconfig.ps1 -action p" command via Powershell as FAST
For SharePoint service account. The command completed successfully.
-Registered
ServicePrincipalNames for CN=s-fastdevsvc,OU=App
Services,OU=CUSFA,DC=carrier,DC=UTC,DC=com
using SETSPN command.
That’s
it – Issue Resolved. Search results are working as expected.
If
you have any queries/questions regarding the above mentioned information then
please let me know, Thank you.
